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An  adaptive  testing  strategy  is  described  for  use  with  achievement  tests 
which  cover  multiple  content  areas.  The  testing  strategy  combines  adaptive 
item  selection  both  within  and  between  the  subtests  in  the  multiple-subtest 
battery.  A real-data  simulation  was  conducted  in  order  to  compare  the  results 
from  adaptive  testing  with  those  from  conventional  testing,  in  terms  of  test 
information  and  test  length.  Data  for  the  simulation  consisted  of  test 
results  for  365  fire-control  technicians  on  a paper-and-pencll  administratlo 
of  a 232-item  achievement  test  which  was  divided  into  12  subtests,  each 
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covering  a different  content  area.  Correlations  between  subtest  scored  from 
adaptive  and  conventional  testing  were  .90  or  higher  for  eleven  of  the  twelve 
content  areas.  An  information  analysis  showed  that  for  all  12  subtests,  the 
subtest  Information  curves  from  adaptive  testing  were  essentially  Identical  to 
the  corresponding  subtest  Information  curves  from  conventional  testing.  On 
the  average,  the  number  of  items  administered  with  adaptive  testing  was  half 
as  many  as  was  required  with  conventional  testing;  the  shortest  adaptive  test 
battery  used  18%  of  the  total  number  of  items  in  the  conventional  test,  while 
the  longest  used  80%.  The  adaptive  testing  strategy,  therefore,  provided  a 
considerable  reduction  in  test  length  and  virtually  no  loss  In  precision  of 
measurement  when  compared  with  the  conventional  administration  of  the 
achievement  test  battery. 
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An  Adaptive  Testing  Strategy  for  Achievement  Test  Batteries 


Modern  test  theory  (latent  trait  theory)  has  provMed  the  framework  for 
a growing  body  of  research  In  ability  measurement  through  adaptive  testing. 

Weiss  and  Betz  (1973)  have  presented  a comprehensive  review  of  adaptive  testing 
which  suggested  that  adaptive  testing  can  considerably  reduce  testing  time, 
wliile  concurrently  yielding  scores  of  higher  reliability  and  validity  than  those 
yielded  by  conventional  tests.  During  the  past  several  years,  a number  of 
studies  have  been  published  which  were  concerned  with  applications  of  diff- 
erent adaptive  testing  strategies  in  the  ability  domain  (e.g.,  Betz  & Weiss, 

1974,  1975;  Larkin  & Weiss,  1974,  1975;  Lord,  1977;  McBride  & Weiss,  1976; 

Urry,  1977;  Vale  & Weiss,  1975).  Each  of  these  studies,  as  well  as  all  the  pre- 
vious research  in  adaptive  testing  (Weiss  & Betz,  1973),  has  been  concerned  with 

tests  which  covered  only  a single  content  area.  Thus,  all  of  the  branching  pro- 

cedures implemented  for  the  adaptive  selection  of  items  to  be  administered  to 
a testee  have  been  designed  exclusively  for  intra-test  branching.  T1  t is,  items 
were  selected  within  a single,  presumably  unidimensional,  content  area. 

Recent  studies  (e.g.,  Bejar,  Weiss,  £.  Gialluca,  1977;  Bejar,  Weiss,  6 Kingsbury, 
1977)  have  demonstrated  that  unidimensional  approaches  to  intra-test  adaptive 
testing  are  useful  for  measurement  in  the  achievement  domain.  Frequently, 
however,  achievement  tests  span  several  content  areas.  Consequently,  in  many 
cases  the  assumption  of  a single  dimension  may  not  be  appropriate.  For  these 
kinds  of  achievement  tests,  or  for  achievement  test  batteries  covering  a number 
of  separable  content  areas  for  which  separate  scores  are  required,  none  of  the 
existing  ad.iptive  strategies  described  by  Weiss  (1974)  are  directly  applicable. 

There  are  two  reasons  why  many  of  the  adaptive  testing  strategics  developed 
for  single-content  area  ability  tests  may  not  be  appropriate  for  achievement 
tests  which  cover  several  content  areas.  The  first  reason  is  that  although  the 
unidimensional  branching  models  can  be  applied  to  separate  content  areas,  they 
are  not  designed  to  take  into  account  the  information  available  between  content 
areas.  The  second,  and  more  practical,  reason  is  that  it  might  not  be  possible 
to  generate  relatively  large  numbers  of  items,  such  as  those  required  for  many 
adaptive  testing  strategies,  within  one  content  area  in  an  achievement  test. 

Urry  (1977)  has  suggested  that  item  pools  to  be  used  in  adaptive  testing  with 
Owen's  (1975)  Bayesian  testing  strategy  should  include  a minimum  of  100  items 
to  measure  one  dimension.  Although  there  are  no  firm  guidelines  for  other 
adaptive  testing  strategies,  it  is  evident  that  they  will  function  best  with 
large  item  pools.  Thus,  application  of  these  strategies  to  an  achievement  test 
battery  of  five  subtests  would  require  the  test  constructor  to  assemble  500 
items  with  good  psychometric  qualities.  Frequently,  this  is  not  possible. 
Consequently,  in  the  application  of  adaptive  testing  to  the  unique  problems  in 
the  measurement  of  achievement,  an  important  research  Issue  is  the  Identification 
of  adaptive  testing  strategies  which  make  efficient  use  of  existing  item  pools, 
rather  than  requiring  the  re-design  of  test  item  pools  to  meet  the  requirements 
of  specific  adaptive  testing  strategies. 
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The  present  paper  describes  an  adaptive  testing  strategy  which  can  be 
used  in  achievement  tests  with  relatively  small  numbers  of  Items.  The  strategy 
is  designed  for  achievement  test  batteries  or  achievement  tests  with  multiple 
content  areas.  It  Incorporates  both  int ra-subtest  branching  and  Inter-subtest 
branching  in  order  to  efficiently  adapt  the  test  battery  to  each  Individual 
testee.  The  adaptive  testing  strategy  is  applied  to  a test  battery  and  evaluated 
in  terms  of; 

1.  The  reduction  in  number  of  items  administered, 

2.  Correlations  of  ability  estimates  with  those  derived  from  conventional 
administration  of  the  test  battery,  and 

3.  The  effects  of  adaptive  administration  on  the  psychometric  information 
in  the  test  scores. 


METHOD 

Purpose 

The  purpose  of  this  study  was  to  develop  and  evaluate  an  efficient  and 
general izable  adaptive  testing  strategy  for  an  achievement  test  battery  com- 
prised of  a number  of  subtests.  The  adaptive  testing  strategy  developed  is 
designed  to  operate  within  a fixed  item  pool  containing  a relatively  small  number 
of  items  for  each  subtest.  Real  data  simulation  techniques  (Weiss  & Betz,  1973, 
pp.  11-12)  were  used.  That  is,  the  adaptive  testing  strategy  was  applied  to 
item  response  data  obtained  from  the  administration  of  an  achievement  test  battery 
which  had  been  previously  administered  conventionally  by  paper-and-penc 11 . 

Results  for  the  conventional  testing  strategy  were  compared  with  those  for  the 
adaptive  testing  strategy  in  terms  of  both  test  information  and  test  length. 

Procedure 


’^est  I terras  and  Subjects 

Achievement  test  data  were  provided  by  the  Personnel  and  Training  Evaluation 
Program  (PTEP)  of  the  Naval  Guided  Missile  School  at  Dam  Neck,  Virginia.’ 

These  data  were  from  a systems  achievement  test  (SAT  F17603)  battery  administered 
to  365  fire  control  technicians.  The  test  battery  included  twelve  subtests,  each 
covering  knowledge  areas  for  different  equipment  or  subject  matter.  Table  1 
shows  the  content  and  number  of  items  in  each  subtest.  The  test  battery  was 
administered  in  one  booklet  containing  232  items.  The  number  of  items  per  sub- 
test ranged  from  10  to  32;  all  of  the  items  were  multiple-choice  with  four 
response  choices.  The  data  provided  by  PTEP  consisted  of  an  identification 
number  for  each  testee,  the  testee's  number  correct  score  on  each  of  the  twelve 
subtests,  and  correct-incorrect  item  responses  for  each  of  the  232  items. 

I tern  Parameterization 


Items  were  parameterized  using  Urry's  ESTEM  computer  program  (see  Urry, 
1976,  p.  99)  for  latent  trait  item  parameterization  employing  the  three-para- 
meter norma]  ogive  model.  This  program  provided  estimates  of  the  item  discrim- 
ination (a),  item  difficulty  (b) , and  guessing  (c)  parameters.  The  items  for 


’Data  were  generously  supplied  by  Lieutenant  Commander  Lee  J.  Walker  of  PTEP. 
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Table  1 


Number  of  Items  In  Each  Subtest 


Subtest 

Content 

No . of 
Items 

A 

Fire  control  system  casualty 

procedures 

10 

B 

Optical  alignment  group 

10 

C 

Control  console  and  power 

subsystem 

18 

D 

Platform  positioning  equip- 

ment 

22 

E 

Multiplexed  equipment 

18 

F 

Digital  control  computer  and 

software 

18 

C 

Digital  control  computer — 

operator  Interface 

14 

H 

Magnetic  disk  file 

12 

I 

Digital  control  computer — 

missile  interface 

24 

J 

Guidance  and  guidance  testing 

29 

K 

MTRE  MKG  MOD 3 

32 

L 

Spare  guidance  temperature 

monitor 

25 

Total 

232 

each  subtest  were  parameterized  Independently  of  items  in  other  subtests. 

Urry's  item  parameterization  program  calculates  item  parameter  estimates 
using  a two-phase  procedure.  In  the  first  phase,  initial  item  parameter 
estimates  are  determined  for  all  items.  However,  item  parameters  are  not  re- 
ported for  an  item  if  one  or  more  of  the  following  conditions  holds:  1)  a<.HO. 

2)  b>-4.00,  3)  i>4.00,  or  4)  a>.30.  In  the  second  phase,  item  parameters  are 
recomputed  for  all  items  which  are  not  excluded  by  the  criteria  applied  in 
the  first  phase.  In  this  phase,  item  parameter  estimates  are  reported  without 
restrictions  (e.g.,  a may  be  greater  than  .30  for  some  items  in  the  second  phase) 
for  all  items  not  excluded  in  the  first  phase. 

Adaptive  Teeting  Stratsm 

The  adaptive  testing  procedure  was  developed  in  order  to  reduce  to  a min- 
imum the  number  of  items  administered  to  each  individual  with  as  little  Impact 
as  possible  upon  the  measurement  characteristics  of  the  test  battery.  Both 
intra-subtest  adaptive  branching  and  Inter-subtest  adaptive  branching  were  used 
in  the  development  of  the  procedure. 

Intra-Subteet  Branching 

Item  aeleation.  The  basic  concept  for  Intra-subtest  adaptive  branching 
was  that  the  order  in  which  the  items  were  to  be  administered  was  to  be  dependent 
upon  values  of  the  item  information  curve  as  defined  by  Blrnbaum  (1968).  For 
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each  item  in  each  subtest,  item  information  values  were  computed  using 
Equation  1 (Birnbaum,  1968,  p.  462): 

I (6)  = (l-f  (9)]/{ii;£)L  (6)]  + a 4-^  [-DL  (9)]}  [1] 

9 9 9 9 9 9 9 


where 


I 


D • 1.7;  this  is  the  scaling  factor  which  maximizes  agreement  between 
the  normal  ogive  and  logistic  latent  trait  models; 

L (9)  = a (9-i  ); 

9 9 9 

= the  logistic  probability  density  function; 

V = the  cumulative  logistic  distribution  function; 

and  values  for  the  parameters  a , b , and  a . were  derived  for  each  of  the 

9 9 9 

g items  in  a subtest  from  the  results  of  the  item  parameterization  phase. 

The  information  values  for  each  item,  I (9),  were  computed  for  values  of  9 

9 

ranging  from  -3.0  to  +3.0  in  steps  of  .2  for  each  item  in  each  subtest. 

Items  were  selected  within  a subtost  for  each  testee  by  computing  the  value 
of  all  item  information  curves  at  the  current  estimated  achievement  level  (9) 
for  that  testee  using  Equation  1.  The  item  selected  for  administration  was  the 
item  which  had  the  highest  information  value  at  the  testee's  current  level  of 
9.  Once  an  item  was  administered  to  a testee,  it  was  eliminated  from  the  sub- 
test pool  of  available  items  for  that  testee. 

Estimation  of  9.  Owen's  (1975)  Bayesian  scoring  procedure  was  used  for 
this  simulation  study.  This  scoring  procedure  provides  an  achievement  level 
estimate  (9)  after  each  mth  test  item  is  administered.  The  procedure  begins 
with  a prior  estimate  of  O^and  its  variance  • for  the  first  item  of  the 

first  subtest  administered  (rml) , these  were  0.00  and  1.00,  respectively.  An 
item  is  administered  and  scored  as^correct  (1)  or  Incorrect  (0).  For  a correct 
response,  the  revised  estimate  of  9 is  determined  by  Equation  2, 


Vfl 


f(eli) 


9 +(l-c  ) 

m g 


m 


+ o 


HD) 


a +(l-(7  )<t>( 

9 9 


and  its  variance  by  Equation  3, 


[2] 


[3] 
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For  an  Incorrect  response,  the  revised  estimate  of  0 is  determined  by  Equation  4, 


In  Equations  2 through  5 (adapted  from  Owen,  1975,  p.  353) 


[A] 


<t>(D)  is  the  normal  probability  density  function, 
t>(D)  is  the  cumulative  normal  distribution  function. 


D = 


V 


+ a' 


A = c?  + (l-<7  )<t'(-P)  i and 


a , b and  a are  the  item  parameter  estimates. 
•3  9 9 


16) 


[7J 


The  updated  estimates  of  6 from  either  Equations  2 or  4,  along  wltli  their  associated 
variances,  are  used  as  the  prior  estimates  of  9 for  the  selection  of  the  next  test 
item,  which  is  based  on  the  maximum  information  rule  described  above.  The  next 
item  is  administered;  and  a new  value  of  9 is  determined,  which  is  then  used  to 
select  the  next  item.  This  procedure  is  repeated  until  a termination  criterion 
is  reached. 


Termination  ariteria.  Two  criteria  were  used  in  determining  when  administ- 
ration of  items  within  a subtest  should  be  stopped:  1)  when  all  of  the  remain- 
ing items  provided  less  than  a pre-determined  small  amount  of  Information;  or  2) 
when  the  within-subtest  item  pool  was  exhausted.  Testing  was  terminated  for  a 
given  testee  at  the  first  occurrence  of  one  of  these  criteria  within  a given  sub- 
test. In  applying  the  first  criterion,  two  arbitrarily  small  values  of  infor- 
mation were  studied;  testing  was  terminated  when  there  was  no  item  available 
which  provided  an^ Inform;' t ion  value  greater  than  .01  or  .001  at  a given  testee's 
current  level  of  0. 
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Figure  1 dlagramatlcally  summarizes  the  intra-subtest  branching  procedure. 
Appendix  A gives  an  Illustration  of  this  procedure,  using  six  items  from  Subtest  1. 


Figure  1 

Intra-subtest  Branching  Scheme 


Inter-Subtest  Branching 

Subtest  ordering.  The  order  of  administration  for  the  various  subtests 
was  chosen  to  take  maximum  advantage  of  the  intercorrelations  among  them,  thereby 
utilizing  the  redundant  information  in  previously  administered  subtests.  This 
was  accomplished  through  linear  multiple  regression.  First,  the  number  correct 
subtest  scores  for  the  twelve  subtests  were  Intercorrelated , and  the  highest 
bivariate  correlation  was  chosen  from  the  Intercorrelation  matrix.  One  of  these 
two  subtests  was  arbitrarily  designated  to  be  administered  first;  the  other  was 
designated  to  be  administered  second. 

Multiple  correlations  were  then  computed  using  the  subtests  previously 
designated  first  and  second  as  predictor  variables.  Each  of  th > ten  remaining 
subtests,  in  turn,  was  designated  as  the  criterion  variable.  Of  these  ten  sub- 
tests, the  one  which  had  the  highest  multiple  correlation  with  the  first  and 
second  subtests  was  designated  as  the  third  subtest.  This  procedure  was  repeated 
to  select  the  fourth  subtest  for  the  adaptive  administration,  computing  multiple 


I 
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correlat ions  with  the  first  three  subtests  as  predictor  variables  and  each 
of  the  remaining  nine  subtests,  i*-  turn,  as  the  criterion  variable.  That  sub- 
test having  the  highest  multiple  correlation  with  the  first  three  subtests  was 
selected  as  the  fourth  subtest  to  be  administered.  By  adding  one  subtest  to  the 
predictor  set  at  each  subsequent  stage,  this  procedure  was  continued  until  all 
twelve  subtests  were  ordered. 

As  a result  of  this  procedure,  the  order  in  which  the  subtests  were  admin- 
istered was  the  same  for  all  testees.  However,  the  selection  of  items  within 
each  subtest  and  the  order  in  which  those  items  were  administered  varied  with 
testees  as  a function  of  the  amount  of  item  information  provided  at  the  testee's 
current  achievement  estimate. 

Differential  subtest  entru  points.  An  important  feature  of  the  adaptive 
testing  strategy  implemented  in  this  study  was  that  after  the  first  subtest, 
each  testee's  entry  points  for  the  second  and  subsequent  subtests  were  differ- 
entially determined.  For  the  first  subtest,  each  testee's  achievement  level 
was  assumed  to  be  6=0.00.  That  is,  having  no  previous  information  on  which  to 
base  an  estimate  of  the  testee's  achievement  level,  the  initial  item  chosen  from 
the  first  subtest  for  administration  was  the  item  which  provided  the  most  infor- 
mation for  an  estimated  achievement  level  at  the  mean  of  the  6 distribution. 
Thus,  all  testees  began  the  first  subtest  with  the  same  test  item. 

The  entry  point  into  the  item  pool  for  the  second  subtest  was  determined 
from  both  the  examinee's  0 at  the  end  of  the  first  subtest  and  the  bivariate 
regression  of  scores  from  Subtest  1 on  Subtesi  2.  This  regression  equation  was 
based  not  only  on  scores  for  the  items  administered  adaptively,  but  also  on  the 
correlations  derived  from  number  correct  scores  for  all  items  In  each  of  the 
subtests.  The  first  item  to  be  administered  for  a testee  in  the  second  subtest 
was  determined  from  information  provided  by  evaluating  Equation  8. 


2E 


+ A 


[8] 


where 


A 

0^^  is  the  first  6 used  for  selection  of  the  first  test  item  in  Subtest  2, 

/V  A. 

is  the  final  6 for  a testee  at  completion  of  the  adaptive  administration 
of  items  in  Subtest  1, 


Bj2  bivaiiate  regression  coefficient  for  the  regression  of  Subtest  2 

on  Subtest  1,  and 

A is  the  regression  constant. 

The  entry  achievement  level  estimate,  computed  as  6^  by  Equations  2 

and  A, was  used  for  selecting  the  first  item  to  be  administered  in  Subtest  2. 

The  variance  of  this  estimate  (o^  in  Equations  3 and  5)  was  determined  bv 

rr  ■' 

Equation  9,  which  is  the  formula  for  the  squared  standard  error  of  estimate 
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in  bivariate  regression  (adapted  from  Glass  & Stanley,  1970,  p.  143): 


= s: 


2 2 
^12®2 


25iSi2 


,2 

12' 


[9] 


where  the  subscripts  1 and  2 represent  the  first  and  second  subtests. 

Determination  of  the  entry  point  for  the  third  and  subsequent  subtests  was 
merely  a generalization  of  the  method  used  for  the  second  subtest.  The  testee’s 
achievement  level  estimates  from  Subtest  1 (6^^)  and  Subtest  2 (§2)  were  entered  in- 
to the  multiple  regression  equation  for  predicting  Subtest  3 scores  from  scores 
on  Subtests  1 and  2.  This  generated  an  estimated  subtest  score  for  an^individual 

(0„.),  which  was  used  as  the  initial  prior  achievement  level  estimate  (9  ) for  intra- 
b i m 

subtest  branching  in  Subtest  3.  The  squared  standard  error  of  estimate  from  the 

multiple  regression  of  Subtests  1 and  2 on  Subtest  3 was  used  as  the  initial  prior 

variance  (o^)  of  the  Baveslan  achievement  level  estimate  for  Subtest  3.  Figure  2 
m 

illustrates  this  differential  entry  point  procedure. 


t 


Figure  2 

Estimation  of  Initial  Achievement  Level  Estimate  for  Subtest  3 (6,,^) 

J 

From  the  Multiple  Regression  of  Subtest  1 (8j)  and  Subtest  2 (S^) 
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The  inter-subtest  branching  regression  procedure  was  used  for  entry  into 
each  of  the  remaining  subtests.  Each  subsequent  regression  equation  was 
based  on  the  achievement  estimates  from  each  of  the  previously  administered  sub- 
tests. A testee's  achievement  level  estimates  for  each  subtest,  based  on  the 
multiple  regression  of  all  previous  subtests  on  a new  subtest,  was  used  as  the 
initial  Bayesian  prior  0 for  intra-subtest  branching  within  that  subtest.  Item 
selection  and  scoring  within  subsequent  subtests  was  then  based  on  the  intra- 
subtest branching  procedures  described  earlier. 

Conventional  Tes t 


A conventional  test  was  used  for  comparison  with  the  adaptive  testing  strategy. 
The  subtests  were  administered  in  the  same  order  for  both  the  conventional  and 
adaptive  strategies.  In  the  conventional  strategy,  all  items  within  each  sub- 
test were  administered  sequentially,  so  that  all  testees  took  the  same  items  in 
the  same  order.  Hence,  there  was  no  differential  entry  for  the  conventional 
strategy.  In  addition,  all  testees  completed  all  items,  which  is  typical  in 
conventional  testing. 

In  order  to  facilitate  comparison  of  results  with  the  adaptive  strategy, 
Bayesian  scoring  was  employed  for  the  conventional  test.  A mean  of  0.0  and  a 
variance  of  1.0  were  used  as  the  initial  prior  achievement  estimate  of  the 
Bayesian  score  for  each  subtest. 


Data  Analysis 


The  basic  question  examined  in  this  study  was  whether  the  number  of  items 
administered  could  be  reduced  through  adaptive  testing  without  significantly 
changing  the  characteristics  of  the  test  scores.  The  effects  of  reducing  the 
number  of  items  by  the  adaptive  testing  item  selection  procedure  were  evaluated 
by  means  of  both  a correlational  analysis  and  an  information  analysis. 

Covvelation  Anali^sis 

Early  research  comparing  single  test  adaptive  testing  strategies  with 
conventional  testing  strategies  (See  Betz  & Weiss,  1973,  1974;  Larkin  & Weiss, 
1974,  1975;  Vale  & Weiss,  1975;  Weiss,  1973)  demonstrated  that  adaptive  tests 
resulted  in  test  scores  highly  correlated  with  conventional  test  scores,  even 
though  the  adaptive  tests  required  substantially  fewer  items.  Consequently,  in 
the  present  study  Pearson  product-moment  correlations  were  computed  between  sub- 
test achievement  level  estimates  (9)  from  the  conventional  and  adaptive  testing 
procedures  in  order  to  examine  the  extent  of  the  relationship  between  the  scores. 
These  were  computed  separately  for  each  of  the  twelve  subtests.  High  correla- 
tions between  the  scores  would  suggest  that  the  tests  ranked  the  examinees  in 
a similar  order  along  the  achievement  continuum. 

Information  Anal^aie 

Information  analyses  were  conducted  in  order  to  compare  the  adaptive  and 
conventional  testing  strategies  as  a function  of  achievement  levels.  Test  in- 
formation values  for  different  testing  strategies  at  different  levels  on  the 
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.ichievement  continuum  provide  an  indication  of  their  relative  degree  of  precision 
of  measurement  (Birnbaum,  1968). 

Estimated  test  information  curves  were  generated  separately  for  each  subtest 
for  both  conventional  and  adaptive  testing  strategies.  In  the  conventional  test- 
ing strategy,  an  examinee's  subtest  information  value  was  computed  by  summing 
the  item  information  values  at  the  examinee's  final  estimated  achlevr-ment  level 
(0)  for  that  subtest.  An  estimated  information  curve  was  plotted  for  the  total 
group  of  examinees  from  their  Individual  achievement  level  estimates  and  corres- 
ponding Information  values.  For  a conventional  test  this  is  equivalent  to  com- 
puting the  test  information  function  using  the  item  parameters  a,  h,  and  o , as 
suggested  by  Birnbaum  (1968,  pp.  454-464). 

Estimated  subtest  information  curves  were  generated  similarly  for  the 
adaptive  testing  strategy.  The  estimated  value  of  test  information  was  computed 
at  each  testee's  final  achievement  estimate  for  the  subtest  by  summing  the  infor- 
mation values  at  that  9 for  the  particular  subset  of  items  administered  to  that 
testee.  Thus,  for  both  adaptive  and  conventional  testing,  each  test  information 
value  was  computed  at  the  final  value  of  0 for  the  subtest,  based  on  the  infor- 
mation provided  by  the  items  actually  administered. 


RESULTS 


Preliminary  Results 

Item  parameterization.  Table  2 presents  means  and  standard  deviations 
for  estimates  of  the  latent  trait  item  discrimination  (a),  difficulty  (b) , and 
guessing  (c*)  parameters  for  the  items  in  the  twelve  subtests.  Complete  distri- 
butions of  individual  item  parameter  estimates  by  subtest  are  shown  in  Appendix 
Table  B-1. 


■'’able  2 

Means  and  Standard  Deviations  of  Normal  Ogive  Item  Discrimination  (a). 


Difficulty  (b) , ajid  Guessing  (a)  Parameters  for  12  Subtests 


Subtest 

Number 

Avail- 

able 

of  It ems 
Parame- 
terized 

a 

b 

O 

— 

Mean  S 

.D. 

Mean 

S.D. 

Mean 

S.D. 

A 

10 

10 

1.90 

.62 

.06 

1.03 

.52 

.11 

B 

10 

10 

2.12 

.86 

.31 

1.29 

.53 

.18 

C 

18 

15 

1.80 

.56 

.54 

1.30 

.55 

.08 

D 

22 

19 

1.60 

.60 

.43 

1.28 

.1*1 

.08 

E 

18 

17 

1.57 

.65 

.74 

1.32 

.1*1 

.10 

F 

18 

18 

1.58 

.43 

1.19 

1.45 

.56 

.09 

G 

14 

13 

1.98 

.94 

1.26 

. 52 

.18 

H 

12 

12 

2.12 

.90 

.84 

1.10 

.43 

.10 

I 

24 

22 

1.49 

.59 

.88 

1.36 

.43 

.10 

J 

29 

23 

1.66 

.57 

1.28 

1.12 

.44 

.14 

K 

32 

24 

1.48 

.61 

.91 

1.39 

.43 

.14 

L 

25 

18 

1.73 

.58 

1.44 

1.34 

.52 

.17 
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From  the  total  item  pool  of  232  items,  item  parameter  estimates  were 
obtained  for  201  items  (87%).  Several  of  the  subtests  (A,  B,  F,  H)  did  not 
lose  any  items  in  the  calibration  process;  the  largest  loss  (28%  of  the  original 
number  of  items)  occurred  for  Subtest  L. 

Mean  item  discrimination  (a)  ranged  from  1.A8  for  Subtest  K to  2.12  for 
Suhtest  H,  while  mean  item  difficulty  ranged  from  .06  for  Subtest  A to  1.44 
for  Subtest  L.  Mean  estimates  for  the  a parameters  of  these  four-alternative 
multiple  choice  items  were  relatively  high,  ranging  from  a low  of  .43  to  a high 
of  .36. 

Subtest  ordcrinfi.  Table  3 shows  the  product -moment  intercorrelations  among 
subtest  scores  for  the  twelve  content  area  subtests  used  to  determine  the  order 
in  which  the  subtests  would  he  administered  in  the  adaptive  test.  The  highest 
bivariate  correlation  (.53)  was  between  content  areas  C and  K,  which  were  desig- 
nated Subtest  1 and  Subtest  2,  respectively. 


Table  3 


Intercorrelations  Among  Content  Area  Scores 


A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

B 

31 

C 

40 

37 

D 

36 

40 

46 

E 

37 

37 

48 

38 

F 

30 

26 

36 

39 

38 

C 

30 

38 

41 

36 

46 

35 

H 

25 

29 

29 

28 

35 

30 

36 

1 

23 

33 

42 

48 

47 

45 

41 

28 

J 

19 

35 

27 

33 

28 

33 

33 

27 

40 

K 

42 

33 

53 

39 

41 

30 

37 

28 

35 

27 

L 

27 

27 

22 

14 

29 

16 

27 

26 

26 

31 

26 

Note . Decimal  points  omitted. 


Table  4 contains  the  multiple  correlations  for  each  subtest  predicted  from 
all  previous  subtests  and  shows  the  ordering  of  subtests  based  on  the  multiple 
correlations.  The  second  column  of  Table  4 shows  the  order  sequence  numbers 
for  the  tests,  based  on  their  ordering  by  the  multiple  corr.'  latlon  procedure. 
These  order  sequence  numbers  are  used  throughout  the  remainder  of  this  report  to 
Identify  the  subtests.  The  multiple  correlations  reported  In  Table  4 ranged  from 
a low  of  .22,  for  predicting  the  score  on  Subtest  L (12)  from  the  score  on  Sub- 
test C (1),  to  a high  of  .57,  for  predicting  performance  on  Subtest  D (5)  from 
performance  on  the  best  weighted  linear  combination  of  Subtests  C,  K,  E,  I 
(1.2, 3, 4). 

The  Inter-subtest  multiple  correlations  shown  in  Table  4 were  not  high 
enough  to  justify  applying  a unidimensional  adaptive  testing  strategy  model 
across  subtests;  Instead,  a mul tl-subtest  branching  strategy  was  developed  and 
Implemented  as  a more  appropriate  procedure  for  this  achievement  test  battery. 
Appendix  Table  B-2  shows  the  raw  score  regression  weights  for  the  regression 
equations  used  In  determining  differential  entry  level  achievement  estimates, 

for  each  subtest  subsequent  to  the  first, 
c 
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Table  4 

Multiple  Correlations  AmonR  Ordered  Subtests 


Criterion 
Subtest  Order 

Predictor  Subtest 

C 

K 

E 

I 

D 

G 

F 

A 

B 

J 

H 

C 

1 

K 

2 

53 

E 

3 

48 

51'" 

I 

4 

42 

45 

53 

D 

5 

46 

49 

52 

'7 

G 

6 

41 

45 

52 

55 

55 

F 

7 

36 

38 

44 

51 

53 

53 

A 

8 

40 

47 

49 

49 

51 

52 

52 

B 

9 

37 

40 

44 

46 

49 

51 

51 

52 

I 

10 

27 

31 

34 

43 

45 

46 

47 

47 

50 

H 

11 

29 

32 

39 

40 

41 

44 

45 

45 

46 

46 

L 

12 

22 

27 

33 

35 

35 

36 

37 

39 

41 

44 

45 

Note . Decimal  points  omitted. 


Value  for  R , the  multiple  correlation  of  Subtest  E, predicted  from 
E • CK 

Subtests  C and  K. 


Conparison  of  Adaptive  and  Conventional  Teste 

Test  length.  The  number  of  items  administered  under  both  the  adaptive  and 
conventional  test  strategies  is  summarized  in  Table  5.  Appendix  Table  B-3  pro- 
vides the  frequency  distribution  of  number  of  items  administered  by  the  adaptive 
testing  strategy  for  each  of  the  twelve  subtests,  and  Table  B-4  gives  this  fre- 
quency distribution  for  all  subtests  combined. 


Table  5 


Number  of  Items  Administered  in  12  Adaptive  and  Conventional  Subtests 


Subtest 

Conventional 

Test 

Mean 

Adapt ive 

S.D. 

Test 

M<n 

Range 

Max 

Percent 
Reduct  ion 

1 

15 

8,73 

1.86 

4 

13 

41.8 

2 

24 

14.12 

2.90 

4 

20 

41.2 

3 

17 

9.87 

3.  38 

2 

17 

41,9 

4 

22 

12.57 

4.60 

2 

22 

42.9 

5 

19 

11.55 

3.58 

1 

18 

39.2 

6 

13 

4.70 

2.10 

1 

12 

63.8 

7 

18 

7.44 

3.21 

1 

15 

58.7 

8 

10 

7.07 

1.71 

1 

10 

29.3 

9 

10 

6.44 

1,72 

1 

9 

35.6 

10 

23 

8.42 

5.54 

1 

22 

63.4 

11 

12 

5.52 

2.97 

1 

12 

54.0 

12 

18 

5.41 

3.20 

1 

15 

69.9 

Mean 

16.  75 

8.49 

3.06 

1.67 

15.42 

49.3 

Test  Battery 

201 

101.84 

24.08 

27 

153 

49.3 

Computed  by  the  formula  100-[(Mean  number  of  items  in  adaptive  test/mean 
number  of  items  in  conventional  test)xlOO] 
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The  data  in  Table  5 show  substantial  reductions  in  test  length  as  a result 
of  the  adaptive  testing  strategy.  For  Subtest  1,  15  items  were  administered 
by  the  conventional  procedure  while  from  4 to  13  items  were  administered  by 
the  adaptive  procedure.  Fifty  percent  of  the  group  answered  between  7 and  10 
items  (see  Table  B-3).  The  mean  number  of  items  administered  by  the  adaptive 
strategy  in  Subtest  1 was  8. 73. which  represents  a 41.8%  reduction  from  the  number 
of  items  required  by  the  conventional  test. 

Similar  results  were  observed  for  the  other  subtests.  Reduction  of  number 
of  items  required  by  the  adaptive  test  varied  from  a low  of  29.3%  for  Subtest 
8 to  a high  of  69.9%  for  Subtest  12,  in  which  a mean  of  5.41  items  was  admini- 
stered by  the  adaptive  strategy.  In  Subtest  12,  between  3 and  7 items  were 
administered  to  50%  of  the  testees  In  the  adaptive  strategy  as  compared  to  18 
items  for  each  testee  in  the  conventional  test.  Subtest  12  had  the  highest 
percent  reduction.  In  all  probability,  this  was  attributable  to  the  Increased 
accuracy  of  the  test  entry  point  from  the  multiple  regression  of  the  scores  on 
the  eleven  prior  subtests. 

It  is  interesting  to  note  that  for  Subtests  5 through  12,  the  minimum  number 
of  items  administered  by  the  adaptive  procedure  was  one.  Table  B-3  shows  that 
for  several  of  these  subtests,  a relatively  substantial  number  of  testees  was 
administered  only  one  item,  l.e.,  almost  10%  of  the  total  group  for  Subtests  6, 

11, and  12.  The  minimum  number  of  items  administered  by  the  adaptive  strategy 
was  less  for  tests  later  in  the  adaptive  testing  sequence.  This  probably  re- 
sulted from  the  increased  use  of  prior  test  information  for  determining  the 
initial  item  to  be  administered. 

Although  minimum  numbers  of  items  were  administered  at  relatively  high  fre- 
quencies by  the  adaptive  strategy,  the  maximum  numbers  of  items  were  administered 
to  very  few  testees  (Table  B-3).  For  Subtests  3,  4,  8,  and  11  the  maximum  number 
of  items  administered  by  the  adaptive  strategy  was  the  same  as  that  administered 
by  the  conventional  test;  frequencies  associated  with  these  maximums  were  2,  1, 

5,  and  1,  respectively.  For  the  remaining  eight  subtests,  none  of  the  testees 
received  the  same  number  of  items  in  the  adaptive  tests  as  they  did  in  the  con- 
ventional test. 

The  conventional  test  battery  consisted  of  201  items  administered  to  all 
testees.  The  average  number  of  items  administered  by  the  adaptive  strategy 
(see  Table  5)  was  101.84,  representing  a 49.3%  reduction  in  number  of  items 
administered.  The  median  number  of  items  administered  was  103  (see  Table  B-4) , 
indicating  a slight  negative  skew  to  the  distribution.  Fifty  percent  of  the 
testees  received  between  86  and  119  items  in  the  adaptive  battery,  representing 
reductions  of  57.2%  to  40.8%  for  half  of  the  testees.  As  Table  B-4  shows,  none 
of  the  testees  required  all  the  items  in  the  adaptive  administration.  The 
longest  adaptive  battery  administered  required  153  items  for  one  testee,  repre- 
senting a 23.9%  reduction  in  test  length;  the  shortest  adaptive  battery  for  one 
testee  required  only  27  items,  representing  a test  length  reduction  of  86.6%. 

Corvelation  Analneie 

Table  6 shows  the  Pearson  product-moment  correlation  of  the  Bayesian 
achievement  level  estimates  (0)  for  the  conventional  and  adaptive  testing  stra- 
tegies. Eleven  of  the  twelve  correlations  were  greater  than  .90.  The  highest 
correlations  were  .98  for  Subtests  2 and  8;  the  lowest  was  .74  for  Subtest  6. 
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Table  6 

Correlation  (r)  of  Bayesian  Achievement  Level  Estimates  (0) 
For  the  Adaptive  and  Conventional  Testing  Strategies  by  Subtest 


Subtest 

No. 

I terns 

r 

Cronbach' s 
Alpha 

1 

15 

.91 

.57 

2 

24 

.98 

.69 

3 

17 

.96 

.54 

4 

22 

.97 

.65 

5 

19 

.93 

.59 

6 

13 

.74 

.44 

7 

18 

.90 

. 50 

8 

10 

.98 

. 56 

9 

10 

.95 

.39 

10 

23 

.92 

.61 

11 

12 

.91 

. 51 

12 

18 

.94 

.40 

The  items  contributing  to  the  Bayesian  subtest  achievement  level  estimates 
in  the  adaptive  test  were  a subset  of  those  used  in  the  conventional  test. 

Thus,  to  some  extent,  the  magnitudes  of  the  correlations  in  Table  6 were  a 
function  of  this  part-whole  relationship.  This  is  supported  by  a comparison 
with  the  Alpha  internal  consistency  estimates  for  the  conventional  subtests 
shown  in  Table  6.  If  there  were  no  part-whole  relationship,  the  correlations 
between  the  achievement  level  estimates  would  be  restricted  by  tlie  internal  con- 
sistencies. However,  all  the  correlations  were  substantially  higher  than  the 
Alpha  values. 

If  the  magnitude  of  the  correlations  of  the  two  achievement  estimates  were 
primarily  determined  by  the  part-whole  relationship  attributable  to  common  items, 
the  number  of  items  administered  in  a subtest  would  bear  a strong  relationship 
to  these  correlations.  This  was  not  generally  the  case:  One  of  the  two  highest 
correlations  (?**.98)  was  observed  for  Subtest  8,  which  had  only  10  items  in  the 
conventional  test,  while  Subtost  9,  which  also  had  10  items,  had  an  r=.95. 

Although  Subtest  8 had  the  smallest  percentage  reduction  attributable  to  the 
adaptive  administration,  20.3%  (see  Table  5),  Subtest  9 had  a 45. reduction; 
and  Subtest  2 (r*“.98)  had  a 41.7%  reduction.  Subtest  6,  which  had  the  lowest 
f (.74), had  a 63.8%  reduction  attributable  to  adaptive  testing;  but  the  highest 
percent  reduction  (69.9%)  was  observed  for  Subtest  12,  for  which  an  r=.94  was 
observed  between  the  adaptive  and  conventional  achievement  estimates.  Thus, 
these  data  suggest  that  the  magnitudes  of  the  correlations  shown  in  Table  6 
were  not  a direct  function  of  either  the  number  of  items  in  the  conventional 
tests  or  the  internal  consistency  of  those  tests. 

Information  AnaLyeis 

Termination  criterion  .001.  The  first  termination  criterion  Investigated  was 
termination  of  adaptive  testing  when  no  unadmlnlstered  item  providing  an  inform- 
ation value  greater  than  .001  remained  in  the  item  pool  for  the  subtest.  Using 
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this  termination  criterion  with  a possible  15  parameterized  items  in  Subtest  1, 
the  mean  number  of  items  administered  in  the  subtest  was  10.55.  The  smallest 
number  of  items  administered  was  five  items  for  six  testees;  the  largest  number 
administered  was  15  items  for  one  testee.  Of  the  165  testees  4.9%  were  admin- 
istered 10  items. 

Adaptive^test  mean  information  values  [7(6)]  at  intervals  of  estimated 
achievement  (6)  corresponding  to  test  termination  criteria  of  .001  and  .01  are 
shown  in  Table  7.  The  range  of  estimated  achievment  levels  was  essentially  the 
same  for  both  criteria,  although  four  testees  obtained  0 values  in  the  Interval 
2.41  to  2.60  for  the  .001  case.  These  we? e outside  the  range  of  0 values 
obtained  in  the  .01  case. 

For  9 of  the  14  intervals  in  which  at  least  10  testees  were  represented  for 
both  termination  criteria,  no  significant  differences  were  observed  in  mean 
information  values.  Significantly  higher  mean  information  values  were  observed 
for  the  .001  termination  criterion  in  three  intervals  of  0:  0.21  to  0.40, 

0.41  to  0.60,  and  0.61  to  0.80.  For  the  remaining  two  Intervals  in  which 
significant  differences  were  observed,  higher  mean  information  was  observed  for 
the  .01  termination  criterion.  However,  the  differences  in  mean  Information 
were  small,  with  the^largest  mean  difference  in  information  .12  in  the  0.21 
to  0.40  interval  of  0. 

The  strong  similarity  of  the  profiles  resulting  from  the  two  termination 
criteria  for  Subtest  1 and  the  lack  of  any  general  trend  in  direction  of  the 
significant  differences  suggested  that  little  was  to  be  gained  by  use  of  the 
more  stringent  .001  termination  criterion.  Therefore,  the  remainder  of  the 
analyses  were  conducted  with  the  .01  termination  criterion. 

Termination  ariterion  ,0’.  Appendix  Tables  B-5  through  B-16  include 
mean  raw  values  of  estimated  information  (/(O) ] at  intervals  of  0 for  the 
adaptive  and  conventional  tests  for  ordered  Subtests  1 through  12.  These 
values  are  based  on  mean  information  In  test  items  actually  administered  to  each 
testee,  using  the  testee's  0 at  the  termination  of  each  subtest.  Information 
was  computed  at  intervals  of  .02  for  0 ranging  from  +3.0  to  -3.0.  The  values 
in  these  tables  were  smoothed  for  plotting  by  the  method  of  moving  averages, 
averaging  across  three  contiguous  values  with  non-zero  frequencies  in  order  to 
reduce  fluctuations  in  the  mean  information  values  resulting  from  differing  fre- 
quencies and/or  small  frequencies  in  the  intervals  of  0 (McNemar,  1969,  p.  8). 

Figure  3 shows  a plot  of  the  smoothed  information  values  for  Subtest  1; 
the  smoothed  values  for  the  last  subtest  administered,  Subtest  12  , are  shown 
in  Figure  4.  Appendix  Figures  C-1  through  C-10  are  plots  of  smoothed  inform- 
ation values  for  the  remaining  subtests.  For  Subtest  1 the  shape  of  the 
Information  curve  for  the  .ndaptlve  test,  as  shown  in  Figure  3,  was  very  similar 
to  that  for  the  conventional  test.  The  largest  differences  in  smoothed  inform- 
ation values  occurredat  9-- 1.4,  where  the  adaptive  test's  smoothed  information 
value  was  2.54  and  that  of  the  conventional  test  was  2.47,  and  at  9“1.3,  where 
Che  conventional  test's  information  value  was  1.70  and  that  of  Che  adaptive 
test  was  1.93. 
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Table  7 

Adaptive  Test  Mean  Information  Values  [7(0)]  at  Estimated 
Achievement  Levels  (9)  for  Termination  Criteria  of  .001  and  .01  for  Subtest  1 


9 Interval 

Terminal  Ion 
Criterion  .001 

Termination 
Criterion  .01 

Mean  Difference 

N 

7(9) 

S.D. 

N 

7(9) 

S.D. 

Min 

Max 

t 

df 

-3.00 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

0 

0 

-2.39 

-2.20 

0 

0 

-2.19 

-2.00 

0 

0 

-1.99 

-1.80 

0 

0 

-1.79 

-1.60 

10 

.63 

.24 

11 

.70 

.29 

-.60 

19 

-1.59 

-1.40 

19 

* 'rr/2 

.38 

22 

1.85 

.40 

-1.06 

39 

-1. 39 

-1.20 

22 

2.76 

.18 

21 

2.37 

.04 

-2.74** 

41 

-1.19 

-1.00 

29 

2.88 

.04 

23 

2.86 

.04 

1.79 

50 

-0.99 

-0.80 

25 

2.89 

.07 

25 

2.86 

.06 

1.63 

48 

-0.  79 

-0.60 

36 

3.41 

.22 

33 

3.36 

.24 

.90 

67 

-0.59 

-0.40 

21 

4. 19 

.09 

21 

4.15 

.15 

1.05 

40 

-0.  39 

-0.20 

33 

4.20 

.11 

31 

4.21 

.11 

-.36 

62 

-0. 19 

0.00 

27 

3.  72 

.19 

27 

3.72 

.19 

.00 

52 

0.01 

0.20 

27 

3.10 

. 18 

35 

3.02 

.21 

1.58 

60 

0.21 

0.40 

38 

2.55 

.12 

26 

2.43 

.09 

4.33** 

62 

0.41 

0.60 

28 

2.23 

.09 

42 

2.17 

.04 

3.80** 

68 

0.61 

0.80 

23 

1.97 

.06 

14 

1.90 

.00 

4.34** 

35 

0.81 

1.00 

12 

1.81 

.05 

10 

1.85 

.00 

-2.52* 

20 

1.01 

1.20 

5 

1.74 

.00 

13 

1.74 

.00 

1.21 

1.40 

6 

1.86 

.05 

0 

1.41 

1.60 

0 

11 

2.19 

.00 

1.61 

1.80 

4 

2. 34 

.00 

1.81 

2.00 

0 

2.01 

2.20 

0 

2.21 

2.40 

0 

0 

2.41 

2.60 

0 

2 

5.23 

.32 

2.61 

2.80 

0 

0 

2.81 

3.00 

0 

0 

* p<.05 

**  p<,01 


Since  mean  Information  values  were  available  for  both  adaptive  and 
conventional  tests  for  Intervals  of  0,  It  was  possible  to  test  the  statis- 
tical significance  of  the  difference  In  mean  estimated  Information  between 
the  adaptive  and  conventional  strategies.  This  was  done  by  computing  t 
ratios  based  on  the  raw  Information  values  In  Tables  B-5  through  B-16  for 
each  0 Interval  containing  at  least  ten  testees  in  both  the  adaptive  and  con- 
ventional strategies.  Computed  t-ratios  were  based  on  an  independent 
groups  t-tesc.  Although  the  same  testees  were  used  in  determining  informa- 
tion values  for  the  two  testing  strategies,  a repeated  measures  t-test  could 
not  be  used  ^Ince  the  same  testees  did  not  necessarily  fall  Into  the  same 
Interval  of  8 on  both  the  adaptive  and  conventional  tests. 
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Figure  3 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  1 
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Contrasts  on  mean  raw  information  values  provided  by  the  adaptive  and 
conventional  testing  strategies  for  Subtest  1 (see  Table  B-5)  showed  significant 
t ratios  (p<.01)  for  the  § intervals  -1.39  to  -1.00  and  0.41  to  1.00.  The 
adaptive  test  provided  significantly  higher  mean  Information  than  the  conventional 
test  over  the  0 Intervals  -1.39  to  -1.20  and  0.81  to  1.00;  the  conventional 
test  provided  significantly  higher  mean  information  than  the  adaptive  test  for 
the  Intervals  -1.19  to  -1.00  and  0.41  to  0.80.  For  the  remaining  § Intervals, 
there  were  no  statistically  significant  differences  in  mean  information. 


Similar  information  curves  from  the  two  testing  strategies  are  shown  for 
Subtest  12  in  Figure  4.  Throughout  the  common  range  of  §,  the  two  curves  were 
very  similar  in  shape;  however,  where  relatively  large  differences  in  information 
occurred,  the  differences  favored  the  conventional  test.  The  major  exception 
was  at  9-1.5,  where  the  difference  favored  the  adaptive  test.  For  Subtest  12, 
the  adaptive  test  provided  0 values  in  a wider  range,  with  46  of  365  testees 
obtalnlng^G  values  less  than  -1.8  on  the  adaptive  test;  none  of  the  testees 
obtained  0 values  less  than  -1.8  on  the  conventional  test. 

Contrasts  on  mean  raw  Information  values  provided  by  the  adaptive  and 
conventional  testing  strategies  for  Subtest  12  (see  Table  B-16)  showed  one 
significant  t ratio  (p<.05)  for  the  0 Interval  -.99  to  -.80.  In  that  Interval 
the  adaptive  test  provided  significantly  higher  mean  Information  than  the  con- 
ventional test.  For  the  remaining  § Intervals,  there  were  no  statistically 
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Flgure  4 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  12 
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significant  differences  between  the  estimated  Information  values  from  the  adap- 
tive and  conventional  testing  strategies  for  Subtest  12. 


As  shown  in  Tables  B-6  through  B-15  for  Subtests  2 through  11,  the  overall 
trend  was  that  there  were  few  significant  differences  between  the  estimated 
information  values  at  all  0 intervals  where  t-tests  were  computed.  The 
largest  number  of  8 Intervals  for  which  statistically  significant  differences 
in  estimated  information  values  were  obtained  was  b of  a possible  14  contrasts 
for  Subtest  1 (Table  B-5) ; for  that  subtest  two  of  the  differences  favored 
the  adaptive  test  and  four  favored  the  conventional  test.  Two  of  the  subtests 
(3  and  10)  showed  no  statistically  significant  differences  in  mean  estimated 
Information  values  between  conventional  and  adaptive  testing.  The  general  lack 
of  differences  in  the  information  curves  is  reflected  In  the  plots  of  smoothed 
information  values  for  Subtests  2 through  11  shown  in  Appendix  Figures  C-1 
through  C-10. 


Disausaion 

This  paper  has  presented  an  adaptive  testing  strategy  designed  for  use 
with  the  achievement  test  batteries  covering  multiple  content  areas.  One  goal 
of  the  strategy  was  to  select  and  administer  items  within  a subtest  as  a 
function  of  the  amount  of  Information  provided  by  each  item  at  each  testee's 
current  estimated  achievement  level.  A second  goal  was  to  use  redundant  inform- 
ation between  and  among  subtests,  by  predicting  a testee's  performance  on  subsequent 
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sv  tests  based  on  performance  on  previous  subtests,  to  determine  appropriate 
differential  entry  points  in  adaptive  branching  between  subtests.  It  was 
hypothesized  that  attaining  these  goals  In  the  design  of  an  adaptive  testing 
strategy  would  result  in  considerable  reduction  in  the  number  of  Items  adminis- 
tered to  each  testee,  while  sacrificing  little*  If  any*  test  information  com- 
pared to  that  obtainable  by  administering  the  entire  test  battery  conventionally. 
Thus,  the  focus  of  this  adaptive  testing  strategy  is  utilization  of  an  existing 
item  pool  for  an  achievement  test  battery  to  efficiently  measure  or  estimate 
each  testee 's  achievement  level. 

Applicability  of  the  ICC  Model 

In  order  to  Implement  the  adaptive  testing  strategy,  it  is  necessary  to 
first  obtain  item  parameters  using  the  item  characteristic  curve  (ICC)  model. 
These  parameters  are  then  used  to  compute  an  information  curve  for  each  test 
item.  The  item  information  curves  are  used, in  turn,  in  the  process  of  intra- 
test branching. 

The  calibration  of  the  achievement  test  items  used  in  this  study  by  the 
ICC  model  permits  an  opportunity  to  determine  the  applicability  of  that  model 
to  achievement  test  data.  Bejar,  Weiss,  and  Kingsbury  (1977)  specifically 
evaluated  the  applicability  of  the  model  to  a college  classroom  achievement  test. 
They  found  that  78%  of  the  309  items  they  studied  yielded  ICC  item  parameter 
estimates.  In  the  present  study,  87%  of  the  items  submitted  to  Urry's  (1976) 
calibration  procedure  resulted  in  item  parameter  estimates  acceptable  by  Urry's 
criteria. 

Items  were  calibrated  within  content  areas  in  the  present  study,  while  in 
the  Bejar  et  al.  study,  calibration  was  in  the  context  of  the  total  set  of  items. 
Nevertheless,  both  studies  showed  that  the  achievement  test  items  analyzed  had 
sufficiently  high  discrimination  parameters  to  be  useful  in  adaptive  testing. 

In  the  present  study,  the  mean  discrimination  (a)  of  all  the  test  items  was 
1.69;  the  corresponding  value  in  the  Bejar  et  al.  study  was  1.20.  There 
was,  however,  a substantial  difference  in  the  a (guessing)  parameter  between 
the  two  studies.  Although  both  studies  used  multiple-choice  items  with  four 
alternative  answers,  the  mean  value  of  the  a parameter  in  the  Bejar  et  al. 
study  was  .29;  the  mean  value  obtained  in  the  present  study  was  .48. 

There  are  at  least  two  possible  explanations  for  the  higher  a parameter 
estimates  in  the  present  study.  The  first,  and  more  likely,  explanation  is  that 
the  (7  parameter  is  poorly  estimated  by  Urry's  program  with  the  sample  sizes 
and  numbers  of  items  used  in  the  present  study.  As  Gugel,  Schmidt,  and  Urry 
(1976)  show,  the  c parameter  is  very  poorly  estimated  by  Urry's  calibration 
program  for  a minimum  of  50  items  and  500  persons.  Consequently,  when  <? 
parameters  are  estimated  from  data  on  as  few  as  10  items  from  365  persons  » 

(as  in  the  present  study),  it  is  likely  that  there  is  a wide  dlscrepency  between 
the  a parameter  estimates  and  their  true  values.  Thus,  the  high  values  of  the 
a parameter  observed  in  the  present  study  may  have  resulted  from  inadequacies 
of  the  parameter  estimation  procedure. 

A second  possible  explanation  for  the  high  a values  is  that  some  of  the 
distractors  in  these  four-choice  items  do  not  operate  effectively  as  distractors. 
If  this  were  the  case,  a testee  with  an  "infinitely  low  level  of  0"  would  be 
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able  to  eliminate  one  or  more  distractors  and  still  randomly  choose  between  the 
remaining  answers.  This  is  contrary  to  the  concept  of  the  testee  with  an 
"infinitely  low  value  of  9"  used  to  interpret  the  a parameter.  Nevertheless, 
the  possibility  exists  that  if  the  elements  of  the  set  of  distractors  are  not 
all  on  the  same  achievement  dimension,  lilgh  values  of  c may  be  found  In  real  test 
data. 


Intra-Sub  test  Sranahina 


The  intra-subtest  item  selection  procedure  utilized  in  this  study  is  a 
variation  of  the  maximum  likelihood  strategies  of  adaptive  testing  (see  Weiss, 
197A,  pp.  62-66).  Maximum  likelihood  adaptive  testing  strategies  typically 
combine  maximum  likelihood  scoring  with  selection  of  Items  based  on  maximum  item 
information  at  the  testee's  current  value  of  §.  The  present  strategy  differs 
in  that  Bayesian  scoring  was  used  in  place  of  maximum  likelihood  scoring;  the 
maximum  information  item  selection  rule  was  used  as  in  maximum  likelihood  adap- 
tive testing. 

In  developing  the  intra-subtest  branching  scheme,  consideration  was  given 
to  using  maximum  likelihood  procedures  for  scoring  the  items.  However,  given 
the  requirement  In  maximum  likelihood  scoring  of  one  correct  item  response  and 
one  incorrect  item  response  before  a 6 can  be  generated,  it  was  determined  to  be 
unfeasible.  Hence,  the  Bayesian  scoring  approach  was  used  so  that  prior  infor- 
mation could  influence  subsequent  achievement  level  estimates  with  as  few  as  one 
item  administered. 

In  general,  the  use  of  maximum  likelihood  scoring  and  Bayesian  scoring  on 
the  same  data  will  not  give  numerically  identical  results.  Although  scores 
obtained  from  the  two  scoring  methods  are  likely  to  be  highly  correlated,  the 
Bayesian  scoring  method  will  result  in  scores  which  have  a restricted  range 
(Lord,  1976).  This  results  from  the  fact  that  Owen's  (1975)  Bayesian  scoring 
routine  assumes  a normal  prior  distribution  of  6 in  the  population;  the  result 
is  6 estimates  which  are  regressed  toward  the  mean.  The  effect  is  a lack  of  0 
estimates  at  the  high  and  low  ends  of  the  distribution. 

This  restriction  in  range  can  affect  the  present  branching  strategy  for 
testees  whose  true  achievement  levels  are  very  high  or  very  low.  If  there  are 
items  which  provide  information  only  at  the  extremes  of  the  distribution  (i.e., 
very  difficult  or  very  easy  items  of  very  high  discrimination),  it  is  possible 
that  the  regressed  9 estimate  from  the  Bayesian  strategy  will  terminate  testing 
too  soon. 

Future  research  should  address  itself  to  ways  of  eliminating  the  effects 
of  regressed  Bayesian  0 estimates.  One  possible  modification  of  the  testing 
strategy  would  be  to  use  Bayesian  scoring  only  when  a maximum- 1 Ikel ihood  stra- 
tegy is  not  feasible,  i.e.,  after  one  item  has  been  administered  or  when  all 
items  are  answered  correctly  or  incorrectly.  When  these  conditions  do  not  occur, 
maximum  likelihood  scoring  could  then  be  used.  Another  possibility  would  be  to 
use  a Bayesian  scoring  procedure  throughout  the  adaptive  test  administration; 
at  the  termination  of  Item  administration  within  a subtest,  estimated  achieve- 
ment scores  could  then  be  re-computed  using  maximum  likelihood  scoring.  If 
continued  testing  were  relevant,  additional  items  would  be  administered  and 
scored  by  maximum  likelihood  until  additional  items  provided  no  further  infor- 
mation. 
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Inter-Subtest  Branching 

The  procedure  for  determining  entry  points  into  later  tests  in  the  adaptive 
sequence  from  the  data  obtained  from  earlier  tests  was  based  on  a linear  multiple 
regression  of  previously  administered  subtest  scores.  In  order  to  implement 
this  procedure,  however,  it  was  necessary  to  order  the  twelve  subtests  to  obtain 
the  relevant  regression  equations.  The  subtests  were  ordered  by  a procedure 
based  on  stepwise  regression  of  subtest  number  correct  scores,  beginning  with 
the  highest  correlation  in  the  matrix. 

Further  research  is  necessary  to  determine  an  optimal  and  general izable 
procedure  for  ordering  a set  of  subtests  for  adaptive  administration  in  an 
achievement  test  battery.  The  procedure  used  in  this  study  may  be  sub-optimal 
for  several  reasons.  First,  it  was  based  on  subtest  number  correct  scores, 
which  are,  in  themselves,  sub-optimal;  thus,  an  ordering  of  subtests  based  on 
methods  of  subtest  scoring  which  utilize  more  information  about  the  items  and/ 
or  testees  might  result  in  a correlation  matrix  with  different  values.  This 
might  yield  a different  ordering  of  subtests. 

Second,  the  regression  procedure  used  might  lead  to  sub-optimal  test  entry 
points  because  regression  estimates  tend  to  underestimate  extreme  scores.  When 
used  with  more  optimal  .“^coring  methods  (e.g.,  maximum  likelihood  scoring),  this 
characteristic  might  require  the  administration  of  additional  and  unnecessary 
test  items  in  order  to  mitigate  the  effects  of  inappropriate  choice  of  initial 
items.  Third,  ina;*propriate  ordering  of  tests  might  also  result  from  the  ten- 
dency of  stepwise  procedures  to  capitalize  on  characteristics  of  the  data  which 
are  unique  to  a given  sample.  Thus,  a relevant  question  for  future  research  on 
procedures  for  subtest  ordering  is:  given  application  of  the  same  subtest 
ordering  procedure,  whether  or  not  different  subsamples  from  the  same  population 
will  result  in  the  same  subtest  ordering  when  measured  by  the  same  test  battery. 

The  Important  question  to  be  answered  regarding  the  problem  of  inter-subtest 
branching  is  whether  or  not  different  test  ordering  procedures  result  in  different 
orderings  of  subtests.  If  the  answer  were  affirmative,  the  next  question  would 
be  what  effect  ordering  procedures  would  have  both  on  the  number  of  items  admin- 
istered and  on  the  measurement  characteristics  of  the  resultant  achievement 
estimates.  The  necessity  to  order  subtests  in  a test  battery  for  adaptive 
administration  occurs  only  when  all  the  Intercorrelatlon.s  among  the  subtests  are 
neither  zero  nor  1.0.  When  the  subtests  intercorrelate  zero  with  each  other, 
there  is  no  redundant  information  in  scores  on  one  subtest  which  will  be  useful 
in  selecting  the  initial  item  for  subsequent  subtesLs.  At  the  other  extreme, 
if  all  subtests  intercorrelate  perfectly  with  each  other,  the  Information  obtained 
from  one  is  completely  redundant  with  that  obtained  from  any  other;  and  no  further 
testing  is  necessary. 

There  Is  one  other  situation  in  which  it  may  not  be  necessary  to  order  the 
subtests  for  adaptive  administration  of  a test  battery.  This  would  occur  when 
all  the  subtests  in  the  battery  have  equal  correlations  with  each  other.  In 
this  case  the  multiple  correlations  of  each  subtest  with  every  other  subtest 
would  be  equal,  and  each  subtest  would  provide  an  equal  amount  of  redundant  in- 
format ion. 


-22- 


There  are  other  procedures  for  ordering  subtests  which  need  to  be  inves- 
tigated. For  example,  subtests  might  be  ordered  in  terms  of  the  number  of  items 
or  their  reliabilities.  If  subtests  were  ordered  by  number  of  items,  it  would 
seem  logical  to  administer  the  shorter  tests  first,  based  on  the  assumption  that 
as  differential  entry  points  become  more  accurate  due  to  additional  redundant 
information,  the  longer  subtests  would  be  more  useful  later  in  the  battery. 

When  ordering  tests  by  their  reliabilities,  it  would  seem  appropriate  to  admin- 
ister the  more  highly  reliable  subtests  first:  More  accurate  redundant  infor- 
mation would  thus  be  obtained  for  selecting  entry  items  for  later  tests  in  the 
adaptive  sequence.  It  should  be  noted,  however,  that  these  two  criteria  for 
subtest  ordering  may  conflict  with  each  other,  since  subtest  reliabilities  tend 
to  be  higher  for  longer  tests. 

All  subtest  ordering  procedures  discussed  thus  far  result  in  a standard 
ordering  of  subtests  for  all  testees.  However,  if  the  philosophy  of  adaptive 
test  administration  were  applied  to  the  subtest  ordering  problem,  it  would  imply 
that  the  c'l'der  of  subtest  administration  should  vary  for  Individual  testees. 

At  thisstageof  research  in  multidimensional  adaptive  testing,  it  is  not  clear 
how  such  an  individualized  inter-subtest  adaptive  procedure  would  be  implemented. 
It  would  seem  that,  to  some  extent,  adaptive  subtest  selection  would  be  based 
on  the  level  of  test  information  in  the  multivariate  test  space  at  the  indivi- 
dual's levels  of  § upon  completion  of  previous  subtests  in  a battery.  However, 
specific  details  for  the  implementation  of  sucl'  a procedure,  as  well  as  compar- 
isons with  alternative  procedures,  will  have  to  await  future  research. 

CONCLUSIONS 

The  real-data  simulation  study  in  this  report  has  supported  previous  research 
which  demonstrated  that  a typical  achievement  test  can  yield  estimates  of  item 
difficulty  and  discrimination  parameters  useful  for  adaptive  testing.  Thus,  the 
applicability  of  item  characteristic  curve  theory  to  the  measurement  of  achieve- 
ment has  been  further  corroborated. 

An  Important  concern  for  adaptive  testing  using  achievement  test  batteries 
is  whether  or  not  a unidimensional  model  can  be  applied  across  subtests.  The 
inter-subtest  multiple  correlations  obtained  in  the  present  study  were  not  con- 
sidered high  enough  to  warrant  the  application  of  a unidimensional  model  across 
subtests.  Instead,  a multi-content  branching  scheme  was  deemed  appropriate  for 
this  achievement  test  battery. 

The  results  of  this  study  have  shown  that  by  using  this  achievement  test 
battery,  the  amount  of  information  extracted  by  adaptive  testing  closely  approx- 
imated that  for  conventional  testing.  The  number  of  Incidences  of  significant 
differences  between  the  Information  curves  for  the  conventional  and  adaptive 
strategies  was  minimal,  and  there  were  no  significant  differences  in  the  majority 
of  the  information  values  for  the  two  testing  strategies  in  each  of  the  twelve 
subtests.  Given  these  results,  an  obvious  question  regarding  the  administration 
of  achievement  test  batteries  is:  If  a computer  terminal  is  available  for  test 
administration,  why  should  test  time  be  spent  administering  those  test  items 
which  do  not  add  to  the  precision  of  measurement  on  the  test  battery? 
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The  adaptive  testing  strategy  described  in  this  report  provides  methods 
for  Intra-subtest  and  Inter-subtest  branching  which  exclude  the  administration 
of  unnecessary  items.  The  data  indicate  that  on  this  achievement  test 
battery  the  length  of  the  battery  can  be  reduced  by  50%  for  the  typical 
testee.  In  no  case  was  it  necessary  to  administer  in  the  adaptive  battery 
all  of  the  items  Included  in  the  conventional  tests.  Therefore,  adaptive 
testing  can  reduce  the  time  spent  in  testing;  the  time  saved  could  then 
be  used  by  the  testees  for  other  activities,  such  as  additional  instruction. 

It  is  also  possible  that  adaptive  achievement  testing  might  have  positive 
psychological  advantages  (e.g.,  Betz  & Weiss,  1976)i  providing  further 
beneficial  effects  on  the  psychometric  characteristics  of  test  scores. 

At  the  least,  reduced  testing  time  might  result  in  more  favorable  attitudes 
of  the  testees  toward  the  testing  process. 

In  the  adaptive  testing  strategy  Implemented  in  this  study,  test  length 
is  a direct  function  of  the  termination  criterion  employed.  Testing  was 
terminated  within  a subtest  when  none  of  the  remaining  items  had  a 
corresponding  level  of  item  ir^ormatlon  greater  than  .01  (.001  for  Subtest  1) 
at  the  testee's  current  estimated  achievement  level;  this  value  was  arbitrarily 
chosen.  More  research  is  needed  to  determine  optimal  termination  criteria. 

That  the  information  curves  resulting  from  the  adaptive  and  conventional 
strategies  were  found  to  be  highly  correspondent  was  to  be  expected  from 
the  way  in  which  items  were  selected  (based  on  Item  information)  for  the 
adaptive  strategy.  However,  because  of  the  Inapplicability  of  maximum 
likelihood  scoring  in  the  early  stages  of  item  administration  within  a 
subtest,  additional  research  is  needed  to  develop  and  evaluate  optimal 
procedures  for  item  scoring.  In  addition,  further  research  is  needed  for 
identification  and  evaluation  of  optimal  procedures  to  order  subtests  for 
Inter-subtest  branching. 

One  additional  finding  from  the  present  study  was  that  the  adaptive  testing 
strategy  consistently  provided  a wider  range  of  achievement  estimates  than 
did  the  conventional  strategy,  using  the  same  method  for  estimating  0. 

Weiss  (1973)  predicted  that  this  would  occur  in  adaptive  testing.  The  major 
implication  of  this  finding  is  that  adaptive  testing  can  provide  more 
discriminating  measurement  in  the  upper  and  lower  extremes  of  the 
achievement  continuum. 

This  study  has  demonstrated  that  an  adaptive  testing  strategy,  designed 
specifically  for  achievement  test  batteries,  can  substantially  reduce  the 
number  of  items  administered  in  all  subtests  of  the  battery  without  reducing 
the  precision  of  subtest  scores.  The  strategy  appears  to  be  generallzable; 
it  should  be  applicable  to  a variety  of  test  batteries  in  which  there  is 
a fixed  and  relatively  small  subset  of  items  for  each  subtest.  Further 
research  is  needed  to  evaluate  the  performance  of  this  adaptive  testing 
strategy  in  other  test  batteries  and  in  live  testing  situations.  In 
addition,  research  is  needed  to  modify  the  adaptive  testing  strategy  to 
identify  optimal  procedures  for  the  complete  Individualized  administration 
of  an  achievement  test  battery. 
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APPENDIX  A 

Illustration  of  Intra-Subtest  Adaptive  Branching 

The  essential  characteristics  of  the  adaptive  testing  strategy  employed 
In  this  study  have  been  described  In  previous  sections.  However,  to  understand 
the  method  more  completely,  It  Is  helpful  to  see  the  results  of  Its  application 
with  an  actual  testee. 

Figure  A-1  shows  estimated  Item  Information  curves  for  six  Items  from 
Subtest  1.  (There  are  a total  of  15  Items  In  Subtest  1 from  which  only  six 
were  chosen  to  simplify  the  Illustration.)  The  height  of  the  Information  curve 
at  a given  achievement  level  Indicates  the  amount  of  Information  provided  by  the 
Item.  Most  of  the  Items  are  fairly  "peaked";  that  Is,  they  provide  Information 
over  a relatively  narrow  range  of  the  achievement  continuum.  While  the  Infor- 
mation curves  overlap  to  some  degree,  different  Items  provide  different  amounts 
of  Information  at  a given  point  on  the  achievement  continuum.  The  guiding 
principle  for  the  adaptive  procedure  is  to  administer  the  Item  which  provides 
the  most  information  at  the  current  achievement  estimate. 


Figure  A-1 

Estimated  Item  Information  Curves  for  Six  Items  from  Test  1 


Achievement  Level 


^ For  a testee  beginning  Subtest  1,  the  Initial  achievement  estimate  was 
0-0  (this  varied  by  Individual  for  subsequent  subtests);  this  is  shown  by  the 
vertical  dashed  line  In  Figure  A-1.  Of  the  six  Items  In  the  example,  only 
three  Items  had  essentially  non-zero  Information  values  at  §-0;  these  values, 
shown  by  the  horizontal  dotted  lines  in  Figure  A-1, were  .90  for  item  5,  .48 
for  Item  15, and  .04  for  Item  12.  Applying  the  rule  that  thg  item  selected  Is 
the  one  which  provides  the  most  information  at  the  current  0,  item  3 would  be 
selected  for  administration. 


Informat ion 
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Figure  A-2  shows  the  revised  value  of  0=.46  derived  from  the  Bayesian 
scoring  routine,  assuming  that  a correct  answer  was  given  to  Item  5.  The  In- 
formation curve  for  Item  5,  whlch^was  already  administered,  Is  not  shown  In 
Figure  A-2.  At  the  new  value  of  9,  only  Items  15  and  12  provide  non-zero 
values  of  Information.  Since  Item  15  has  an  Information  value  of  .54  and  Item 
12  has  a value  of  .20,  Item  15  Is  selected  as  the  second  Item  to  be  administered 
to  this  testee. 


4 


Figure  A-2 

Estimated  Item  Information  Curves  for  Five  Items  from  Test  1 


Achievement  Level 


Assuming  that  the  testee  had  correctly  answered  item  15,  the_^value  of  9 
Increased  to  .92;  this  is  shown  In  Figure  A-3.  At  that  value  of  0,  Item  12 
provides  .22  information  and  Item  10  provides  .02  information.  Item  12  is 
thus  administered  next.  Assuming  that  Item  12  was  answered  incorrectly,  the 
0 decreased  to  .62,  which  is  plotted  in  Figure  A-4.  The  figure  shows  that  of 
the  three  Items  remaining,  none  provides  any  Information  at  the  current  level 
of  0.  Thus,  there  is  no  need  for  administering  additional  Items  from  Subtest  1, 
and  testing  in  that  subtest  Is  terminated.  The  achievement  level  estimate  of 
0 >.62  Is  taken  as  the  testee's  score  on  Subtest  1,  since  It  is  based  on  all 
items  providing  more  than  non-trlvlal  amounts  of  Information  about  that  testee's 
achievement  level.  For  Inter-test  branching,  0 •■.62  is  used  in  the  regression 
equation  to  determine  the  entry  point  § estimate  for  selecting  the  first  item 
to  be  administered  In  Subtest  2. 
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Ho^.  Dashed  lines  indicate  that  an  item  was  rejected  In  the  first  phase  of  the  Item  parameterUat  icn  procedure. 
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Table  B-2 

Raw  Score  Regression  Weights  (b)  for  Regression  Equations 
Used  to  Determine  Differential  Entry  Points  In  Inter-Subtest  Branchin 
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Table  B-4 

Frequency  and  Cumulative  Percent  of  Total  Number  of  Items  Administered  by 


the  Adaptive  Testing  Strategy 

Across  all 

12  Subtests 

(N-365 

Testees) 

nV. 

Cum 

No. 

Cum 

No. 

Cum 

Items 

Freq. 

Pet. 

Items 

Freq . 

Pet . 

Items 

Freq. 

Pet 

27 

1 

1 

85 

8 

24 

117 

1 

72 

41 

1 

1 

86 

6 

25 

118 

5 

73 

42 

1 

1 

87 

1 

25 

119 

8 

75 

44 

3 

2 

88 

5 

27 

120 

3 

76 

47 

3 

2 

89 

4 

28 

121 

3 

77 

51 

3 

3 

90 

8 

30 

122 

2 

78 

52 

3 

4 

91 

2 

31 

123 

4 

79 

54 

1 

4 

92 

4 

32 

124 

6 

80 

55 

2 

5 

93 

5 

33 

125 

4 

81 

57 

2 

5 

94 

6 

35 

126 

4 

82 

58 

1 

6 

95 

7 

37 

127 

9 

85 

59 

3 

7 

96 

5 

38 

128 

6 

87 

60 

2 

7 

97 

4 

39 

129 

5 

88 

61 

2 

8 

98 

8 

41 

130 

5 

89 

65 

3 

8 

99 

7 

43 

131 

2 

90 

66 

4 

10 

iOO 

5 

45 

132 

4 

91 

68 

1 

10 

101 

8 

47 

133 

4 

92 

69 

2 

10 

102 

4 

48 

134 

2 

93 

70 

4 

12 

103 

6 

50 

135 

3 

93 

71 

2 

12 

104 

4 

51 

136 

3 

94 

72 

1 

12 

105 

6 

52 

137 

1 

95 

73 

1 

13 

106 

8 

55 

138 

2 

95 

74 

4 

14 

107 

2 

55 

139 

5 

96 

75 

2 

14 

108 

11 

58 

141 

2 

97 

76 

5 

16 

109 

11 

61 

142 

2 

98 

77 

1 

16 

no 

6 

63 

144 

2 

98 

78 

3 

17 

111 

6 

64 

145 

2 

99 

79 

2 

17 

112 

8 

67 

146 

1 

99 

80 

5 

19 

113 

6 

68 

147 

1 

99 

81 

I 

19 

114 

4 

69 

148 

1 

99 

82 

2 

19 

115 

4 

70 

149 

1 

99 

83 

4 

21 

116 

4 

72 

153 

1 

100 

84 

3 

21 
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Table  B-5 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(6)] 
and  Mean  Difference  in  Information  and  t Values 
at  Estimated  Achievement  Levels  (9)  for  Subtest  1 


6 Interval 

Adapt ive 

Test 

Conventional 

Test 

Mean  Difference 

Min 

Max 

N 

I (0) 
a 

SD 

N 

SD 

[I  (6)-!  (§)] 
c a 

t 

df 

-3.00 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

0 

0 

-2.39 

-2.20 

0 

0 

-2.19 

-2.00 

0 

0 

-1.99 

-l.bO 

0 

0 

-1.79 

-1.60 

11 

.70 

.29 

14 

.64 

.23 

-.06 

-.58 

23 

-1.59 

-1.40 

22 

1.85 

,40 

23 

1.83 

.35 

-.02 

-.18 

43 

-1.39 

-1.20 

21 

2.87 

.04 

25 

2,73 

.13 

-.14 

-3.49** 

44 

-1.19 

-1.00 

23 

2.86 

.04 

20 

2,89 

.03 

.03 

2.75** 

41 

-0.99 

-0.80 

25 

2.86 

.06 

28 

2.89 

.06 

,03 

1.82 

51 

-0.79 

-0.60 

33 

3.36 

.24 

37 

3.38 

.19 

.02 

.39 

68 

-0.59 

-0.40 

21 

4.15 

.15 

19 

4.15 

.16 

.00 

.00 

38 

-0.39 

-0.20 

31 

4.21 

.11 

24 

4.26 

.06 

.05 

2.01 

53 

-0.19 

0.00 

27 

3.72 

.19 

32 

3,75 

.23 

.03 

.54 

57 

0.01 

0.20 

35 

3.02 

.21 

30 

3.04 

.21 

.02 

.38 

63 

0.21 

0.40 

26 

2.43 

.09 

31 

2.50 

.12 

.07 

2.45* 

55 

0.41 

0.60 

42 

2,17 

.04 

29 

2.23 

.08 

.06 

4.17** 

69 

0.61 

0.80 

14 

1.90 

.00 

27 

1.96 

.07 

.06 

3.19** 

39 

0.81 

1.00 

10 

1.85 

.00 

10 

1.81 

.04 

-.04 

-3.16** 

18 

1.01 

1.20 

13 

1,74 

,00 

7 

1.74 

.01 

.00 

1.21 

1.40 

0 

6 

1.85 

.01 

1.41 

1.60 

11 

2.10 

.00 

3 

2.13 

.00 

-.06 

1.61 

J .80 

0 

1.81 

2.00 

0 

2.01 

2.20 

0 

2.21 

2.40 

0 

2.41 

2.60 

0 

2.61 

2.80 

0 

2.81 

3.00 

0 

*p<.05 

**p<.01 
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Table  B-6 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(0)]  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  2 


9 

Interval 

Adaptive  Test 

Conventional 

Test 

Mean  Difference 

Min 

Max 

N I (§)  S.D. 

a 

N I (§) 

c 

S.D. 

[I  (0)-I  (0)]  t df 

c a 

-3.00 

-2.80 

0 

0 

-2.  7V 

-2.60 

0 

0 

-2.59 

-2.40 

2 

.00 

.00 

0 

-2.39 

-2.20 

16 

.00 

.00 

0 

-2.19 

-2.00 

20 

.01 

.01 

0 

-1.99 

-1.80 

9 

.03 

.02 

0 

-1.79 

-1.60 

7 

.16 

.16 

58 

.32 

.07 

.16 

-1.59 

-1.40 

1 

1.85 

0 

-1.39 

-1.20 

20 

3.50 

.37 

0 

-1.19 

-1.00 

40 

3.14 

.64 

0 

-0.99 

-0.80 

40 

1.60 

.42 

12 

1.32 

.20 

-.28 

-2.22* 

50 

-0.79 

-0.60 

29 

.76 

.36 

30 

.68 

.15 

-.08 

-1.12 

57 

-0.59 

-0.40 

31 

.45 

.01 

49 

.44 

.07 

-.01 

-.79 

78 

-0.39 

-0.20 

33 

.58 

.08 

58 

.63 

.07 

.05 

3.11** 

89 

-0.19 

0.00 

50 

.42 

.53 

80 

.36 

.46 

-.06 

-.68 

128 

0.01 

0.20 

11 

1.44 

.12 

12 

1.24 

.40 

-.20 

-1.59 

21 

0.21 

0.40 

16 

1.82 

.05 

13 

1.81 

.06 

-.01 

-.49 

27 

0.41 

0.60 

15 

1.91 

.01 

23 

1.91 

.01 

.00 

.00 

36 

0.61 

0.80 

5 

1.88 

.01 

6 

1.88 

.02 

.00 

0.81 

1.00 

2 

1.82 

.01 

1 

1.84 

.02 

1.01 

1.20 

0 

0 

1.21 

1.40 

2 

1.73 

2.44 

4 

2.64 

.41 

.91 

1.41 

1.60 

1 

.00 

12 

6.76 

1.50 

6.76 

1.61 

1.80 

1 

.00 

3 

11.56 

3.08 

11.56 

1.81 

2.00 

1 

.00 

1 

15.13 

15.13 

2.01 

2.20 

0 

2 

8.58 

2.82 

2.21 

2.40 

0 

1 

3.44 

2.41 

2.60 

1 

2.55 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

0 

0 

Table  B-7 

Adaptive  and  Conventional  Test  Mean  Infortnatlon  Values  [1(9)]  apd  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  3 


9 

Interval 

Adaptive 

Test 

Conventional 

Test 

Mean  Difference 

Min 

Max 

N 

Ia(9) 

S.D. 

N 

I (9) 
c 

S.D. 

[1  (0)-I  (9)1 
c a 

t 

df 

-3.00 

-2.80 

0 

-2.79 

-2.60 

0 

-2.59 

-2.40 

0 

-2.39 

-2.20 

12 

.00 

.00 

-2.19 

-2.00 

10 

.02 

.01 

-1.99 

-1.80 

27 

.04 

.02 

-1.79 

-1.60 

17 

.14 

.05 

1 

.25 

-1.59 

-1.40 

19 

.49 

.17 

12 

.48 

.13 

-.01 

-.17 

29 

-1.39 

-1.20 

17 

1.03 

.24 

32 

1.14 

.24 

.11 

1.53 

47 

-1.19 

-1.00 

36 

2.05 

.28 

31 

1.97 

.27 

-.08 

-1.19 

65 

-0.99 

-0.80 

15 

2.37 

.66 

28 

2.59 

.05 

.22 

1.77 

41 

-0.79 

-0.60 

42 

2.47 

.06 

40 

2.44 

.08 

-.03 

-1.93 

80 

-0.59 

-0.40 

21 

2.12 

.49 

30 

2.22 

.04 

.10 

1.12 

49 

-0.39 

-0.20 

26 

2.16 

.01 

33 

2.16 

.01 

.00 

.00 

57 

-0.19 

0.00 

42 

.66 

.99 

79 

.89 

1.06 

.23 

1.16 

119 

0.01 

0.20 

15 

2.00 

.55 

26 

1.98 

.58 

-.02 

-.11 

39 

0.21 

0.40 

9 

2.28 

.05 

18 

2.28 

.05 

.00 

.00 

25 

0.41 

0.60 

16 

2.52 

.05 

15 

2.48 

.06 

-.04 

-2.02 

29 

0.61 

0.80 

4 

2.60 

.02 

5 

2.65 

.02 

.05 

0.81 

1.00 

5 

2.08 

1.16 

6 

2.55 

.07 

.47 

1.01 

1.20 

7 

2.29 

.12 

3 

2.28 

.06 

-.01 

1.21 

1.40 

6 

1.86 

.11 

3 

1.83 

.09 

-.03 

1.41 

1.60 

5 

1.22 

.69 

0 

1.61 

1.80 

1 

.00 

1 

.00 

.00 

1.81 

2.00 

1 

.00 

0 

2.01 

2.20 

0 

0 

2.21 

2.40 

0 

0 

2.41 

2.60 

0 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

1 

.00 

0 
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Table  B-8 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(6)]  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  4 


§ 

Interval 

Adaptive  Test 

Conventional  Test 

Mean  Difference 

Min 

Max 

N I (0)  S.D. 

a 

N I (0)  S.D. 

c 

(I  (0)-I  (0)]  t df 

c a 

-3.00 

-2.80 

0 

-2.79 

-2.60 

0 

-2.59 

-2.40 

0 

-2.39 

-2.20 

1 

.00 

-2.19 

-2.00 

6 

.01 

.00 

-1.99 

-1.80 

17 

.03 

.02 

-1.79 

-1.60 

12 

.17 

.07 

3 

.21 

.06 

.04 

-1.59 

-1.40 

18 

.44 

.13 

12 

.49 

.16 

.05 

.94 

28 

-1.39 

-1.20 

25 

1.37 

.37 

20 

1.38 

.32 

.01 

.10 

43 

-1.19 

-1.00 

25 

2.59 

.37 

29 

2.61 

.31 

.02 

.22 

52 

-0.99 

-0.80 

14 

3.87 

.41 

24 

3.63 

.36 

-.24 

-1.88 

36 

-0.79 

-0.60 

23 

5.08 

1.26 

33 

5.06 

.42 

-.02 

-.09 

54 

-0.59 

-0.40 

22 

6.31 

.11 

24 

6.30 

.08 

-.01 

-.36 

44 

-0.39 

-0.20 

21 

5.39 

1.30 

17 

5.70 

.32 

.31 

.96 

36 

-0.19 

0.00 

57 

1.72 

2.04 

81 

1.58 

1.99 

-.14 

-.40 

136 

0.20 

22 

3.21 

.24 

33 

3.17 

.59 

-.04 

-.30 

53 

0.21 

0.40 

6 

2.58 

.10 

10 

2.49 

.00 

-.09 

O.Al 

0.60 

29 

2.19 

.14 

23 

2.20 

.14 

.01 

.26 

50 

0.61 

0.80 

20 

1.57 

.14 

23 

1.71 

.15 

.14 

3.15** 

41 

0.81 

1.00 

15 

1.31 

.09 

14 

1.41 

.01 

.10 

4.13** 

27 

1.20 

5 

.84 

.04 

18 

1.00 

.00 

.16 

1.21 

1.40 

10 

.78 

.01 

0 

1.41 

1.60 

3 

.85 

.07 

0 

1.61 

1.80 

1 

.00 

0 

1.81 

2.00 

2 

.00 

.00 

0 

2.01 

2.20 

1 

.00 

0 

2.21 

2.40 

0 

0 

2.41 

2.60 

0 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

0 

0 

**  p<.01 
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Table  B-9 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(0)]  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  5 


0 

Interval 

Adaptive 

Test 

Conventional 

Test 

Mean 

Difference 

Min 

Max 

N 

I (0) 
a 

S.D. 

N 

I (9) 
c 

S.D. 

(i^(e)-i 

a(9)l  t 

df 

-3.00 

-2.80 

0 

-2.79 

-2.60 

0 

-2.59 

-2.40 

6 

.01 

.00 

-2.39 

-2.20 

4 

.07 

.05 

-2.19 

-2.00 

9 

.27 

.09 

-1.99 

-1.80 

7 

3.31 

.79 

21 

3.48 

.69 

.17 

-1.79 

-1.60 

6 

5.39 

2.66 

10 

4.98 

.38 

-.41 

-1.59 

-1.40 

8 

5.64 

.62 

0 

-1.39 

-1.20 

13 

3.50 

.62 

8 

2.56 

.24 

-.94 

-1.19 

-1.00 

26 

2.04 

.11 

18 

2.02 

.11 

-.02 

-.59 

42 

-0.99 

-0.80 

38 

2.22 

.14 

25 

2.19 

.16 

-.03 

-.79 

61 

-0.79 

-0.60 

25 

2.61 

.07 

33 

2.64 

.06 

.03 

1.76 

56 

-0.59 

-0.40 

33 

2.50 

.45 

29 

2.59 

.08 

.09 

1.06 

60 

-0.39 

-0.20 

34 

2.40 

. .03 

31 

2.40 

.03 

.00 

.00 

63 

-0.19 

0.00 

60 

1.19 

.06 

87 

1.20 

1.26 

.01 

.06 

145 

0.01 

0.20 

21 

2.77 

.08 

25 

2.15 

1.10 

-.62 

-2.57** 

44 

0.21 

0.40 

14 

2.68 

.77 

27 

2.89 

.01 

.21 

1.31 

39 

0.41 

0.60 

10 

2.48 

.80 

16 

2.77 

.09 

.29 

1.45 

24 

0.61 

0.80 

17 

2.39 

. 10 

16 

2.14 

.01 

-.25 

-9.94** 

16 

0.81 

1.00 

2 

1.89 

.13 

0 

1.01 

1.20 

6 

1.34 

.07 

18 

1.59 

.02 

.25 

1.21 

1.40 

8 

1.20 

.01 

0 

1.41 

1.60 

4 

1.27 

.03 

0 

1.61 

1.80 

0 

0 

1.81 

2.00 

1 

.00 

0 

2.01 

2.20 

1 

.00 

1 

.00 

.00 

2.21 

2.40 

1 

.00 

0 

2.41 

2.60 

0 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

1 

A. 

.00 

0 

**  p<.01 
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Table  B-10 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(9)]  and  Mean  Difference 
in  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  6 


9 Interval 

Adaptive  Test 

Conventional  Test 

Mean  Difference 

Min 

Max 

N 

I (9) 
a 

S.D. 

N 

I (9) 
c 

S.D. 

[I  (0)-! 

c a 

< CD 

df 

-3.00 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

4 

.00 

.00 

0 

-2.39 

-2.20 

10 

.00 

.00 

0 

-2.19 

-2.00 

10 

.00 

.00 

0 

-1.99 

-1.80 

21 

.02 

.01 

0 

-1.79 

-1.60 

21 

.11 

.03 

4 

.15 

.03 

.04 

-1.59 

-1.40 

19 

.39 

.13 

32 

.38 

.11 

-.01 

-.29 

49 

-1.39 

-1.20 

35 

.83 

.11 

6 

.94 

.16 

.11 

-1.19 

-1.00 

31 

1.10 

.02 

26 

1.11 

.01 

.01 

2.32* 

55 

-0.99 

-0.80 

16 

1.08 

.01 

32 

1.09 

.01 

,01 

3.27** 

46 

-0.79 

-0.60 

16 

1.21 

.05 

26 

1.26 

.06 

.05 

2.79** 

40 

-0.59 

-0.40 

43 

1.55 

.11 

42 

1.57 

.15 

.02 

.70 

83 

-0.39 

-0.  20 

8 

1.78 

.74 

35 

2.10 

.19 

.32 

-0.19 

0.00 

52 

1.08 

1.22 

75 

.99 

1.33 

-.09 

-.39 

125 

0.01 

0.20 

18 

2.73 

.99 

20 

2.93 

.69 

.20 

.73 

36 

0.21 

0.40 

11 

3.01 

.07 

26 

3.01 

.07 

.00 

.00 

^5 

0.41 

0.60 

10 

2.51 

.89 

14 

2.59 

.75 

.08 

.24 

22 

0.61 

0.80 

11 

2.65 

.03 

10 

2.67 

.03 

.02 

1.53 

19 

0.81 

1.00 

4 

2.96 

.18 

7 

2.94 

.15 

-.02 

1.01 

1.20 

4 

3.60 

.37 

1 

3.52 

-.08 

1.21 

1.40 

2 

4.57 

.63 

3 

4.55 

.41 

-.02 

1.41 

1.60 

1 

5.70 

4 

5.78 

.31 

.08 

1.61 

1.80 

3 

6.54 

.25 

0 

1.81 

2.00 

2 

8.10 

.29 

0 

2.01 

2.20 

1 

.00 

0 

2.21 

2.40 

1 

.00 

0 

2.41 

2.60 

1 

.00 

0 

2.61 

2.80 

0 

* 

1 

10.57 

2.81 

3.00 

0 

0 

5 

1 
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Table  B-11 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(0)]  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  7 


yv 

0 Interval 

Adaptive  Test 

Conventional 

Test 

Mean 

Difference 

Min 

Max 

N 

I (0) 
a 

S.D. 

N 

I (6) 
c 

S.D.  [ 

I (0)-!  (6)]  t 

c a 

df 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

0 

0 

-2.39 

-2.20 

9 

.00 

.00 

0 

-2.19 

-2.00 

31 

.00 

.00 

0 

-1.99 

-1.80 

18 

.00 

.00 

0 

-1.79 

-1.60 

6 

.01 

.01 

0 

-1.59 

-1.40 

19 

.09 

.07 

14 

.09 

.01 

.00 

.00 

31 

-1.39 

-1.20 

15 

.80 

.29 

35 

.62 

.20 

-.18 

-2.54* 

48 

-1.19 

-1.00 

34 

2.42 

.71 

38 

2.14 

.39 

-.28 

-2.10 

70 

-0.99 

-0.80 

26 

4.15 

.21 

0 

-0.79 

-0.60 

24 

3.15 

.40 

35 

3.42 

.39 

.27 

2.59* 

57 

-0.59 

-0.40 

47 

2.12 

.29 

32 

2.13 

.33 

.01 

.14 

77 

-0.39 

-0.20 

17 

1.55 

.04 

43 

1.55 

..04 

.00 

.00 

58 

-0.19 

0.00 

40 

1.00 

4.01 

90 

.78 

.86 

-.22 

-.50 

128 

0.01 

0.20 

9 

2.28 

.19 

21 

2.17 

.57 

-.11 

-.56 

28 

0.21 

0.40 

16 

2.95 

.85 

11 

3.33 

.32 

.38 

1.41 

25 

0.41 

0.60 

10 

4.53 

.26 

16 

4.56 

.20 

.03 

.33 

24 

0.61 

0.80 

8 

4.32 

1.74 

12 

4.94 

.03 

.62 

0.81 

1.00 

9 

4.22 

1.59 

6 

4.67 

.03 

.45 

1.01 

1.20 

2 

4.67 

.03 

4 

4.66 

.04 

-.01 

1.21 

1.40 

2 

5.23 

.02 

2 

5.27 

.10 

.04 

1.41 

1.60 

5 

5.28 

.10 

2 

5.29 

.10 

.01 

1.61 

1.80 

1 

4.30 

0 

1.81 

2.00 

2 

2.96 

.61 

2 

3.55 

.00 

.59 

2.01 

2.20 

1 

2.40 

1 

1.87 

2.21 

2.40 

1 

,00 

0 

2.41 

2.60 

1 

.00 

0 

2.61 

2.80 

1 

.00 

0 

2.81 

3.00 

1 

2.19 

0 

* p<.05 
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Table  B-12 

Adaptive  and  Conventional  Test  Mean  Information  Values  (1(0)]  apd  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (9)  for  Subtest  8 


0 

Interval 

Adaptive 

Test 

Conventional 

Test 

Mean 

Difference 

Min 

Max 

N 

I (6) 
a 

S.D. 

N 

I (0) 

C 

S.D. 

[I  (0)-i 

c a 

(e)]  t 

df 

-3.00 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

0 

0 

-2.39 

-2.20 

8 

.00 

.00 

0 

-2.19 

-2.00 

9 

.00 

.00 

1 

.00 

-1.99 

-1.80 

19 

.00 

.00 

0 

-1.79 

-1.60 

32 

.01 

.01 

0 

-1.59 

-1.40 

61 

.03 

.01 

4 

.05 

.01 

.02 

-1.39 

-1.20 

17 

.11 

.04 

26 

.14 

.03 

.03 

2.81** 

41 

-1.19 

-1.00 

38 

.31 

.05 

29 

.29 

.07 

-.02 

-1.36 

65 

-0.99 

-0.80 

26 

.54 

.08 

43 

.54 

.08 

.00 

.00 

67 

-0.79 

-0.60 

25 

.89 

.09 

59 

.84 

.10 

-.05 

-2.16 

82 

-0.59 

-0.40 

10 

1.26 

.16 

34 

1.26 

.15 

.00 

.00 

42 

-0.39 

-0.20 

18 

1.76 

.12 

37 

1.72 

.13 

-.04 

-1.10 

53 

-0.19 

0.00 

41 

.58 

.92 

70 

.63 

.97 

.05 

.27 

109 

0.01 

0.20 

10 

2.08 

.73 

26 

2.24 

.46 

.16 

.79 

34 

0.21 

0.40 

13 

2.56 

.07 

10 

2.67 

.08 

.11 

3.51** 

21 

0.41 

0.60 

3 

1.88 

1.62 

11 

2.83 

.04 

.95 

0.61 

0.80 

7 

2.78 

.07 

2 

2.82 

.06 

.04 

0.81 

1.00 

6 

1.74 

1.35 

6 

2.58 

.10 

.84 

1.01 

1.20 

3 

2.27 

.06 

4 

2.35 

.05 

.08 

1.21 

1.40 

0 

2 

2.18 

.01 

1.41 

1.60 

2 

2.23 

.16 

1 

2.18 

.05 

1.61 

1.80 

1 

4.35 

0 

1.81 

2.00 

1 

.00 

0 

2.01 

2.20 

1 

9.71 

0 

2.21 

2.40 

0 

0 

2.41 

2.60 

1 

.00 

0 

2.61 

2.80 

1 

.00 

0 

2.81 

3.00 

1 

.00 

0 

**  p<.01 
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Table  B-13 

Adaptive  and  Conventional  Test  Mean  Information  Values  [1(6)]  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (6)  for  Subtest  9 


S Interval 

Adaptive  Test 

Conventional  Test 

Mean  Difference 

Min 

Max 

N 

I (0) 
a 

S.D. 

N 

1^6) 

S.D.  [I  (6)-I  (0)1  t 

c a 

df 

-3.00 

-2.80 

0 

-2.79 

-2.60 

0 

-2.59 

-2.40 

0 

-2.39 

-2.20 

0 

-2.19 

-2.00 

1 

.00 

-1.99 

-1.80 

0 

-1.79 

-1.60 

12 

.64 

.34 

14 

.64 

.23 

.00 

.00 

24 

-1.59 

-1.40 

22 

1.85 

.40 

23 

1.83 

.35 

-.02 

-.18 

43 

-1.39 

-1.20 

20 

2.86 

.04 

24 

2.73 

.18 

-.13 

-3.16** 

42 

-1.19 

-1.00 

24 

2.74 

.59 

20 

2.89 

.03 

.15 

1.13 

42 

-0.99 

-0.80 

25 

2.86 

.06 

28 

2.89 

.06 

.03 

1.82 

51 

-0.79 

-0.60 

31 

3.35 

.24 

34 

3.39 

.20 

.04 

.73 

63 

-0.59 

-0.40 

20 

4.14 

.15 

18 

4.14 

.16 

.00 

.00 

36 

-0.39 

-0.20 

29 

4.21 

.11 

22 

4.25 

.07 

.04 

1.49 

49 

-0.19 

0.00 

50 

1.67 

1.87 

75 

1.36 

1.83 

-.31 

-.92 

123 

0.01 

0.20 

30 

3.03 

.21 

27 

2.90 

.62 

-.13 

-1.08 

55 

0.21 

0.40 

22 

2.32 

.53 

24 

2.50 

.12 

.18 

1.62 

44 

0.41 

0.60 

33 

2.17 

.04 

23 

2.23 

.08 

.06 

3.70** 

54 

0.61 

0.80 

9 

1.69 

.63 

19 

1.97 

.07 

.28 

0.81 

1.00 

8 

1.85 

.00 

5 

1.82 

.05 

-.03 

1.01 

1.20 

7 

1.49 

.66 

4 

1.74 

.00 

.25 

1.21 

1.40 

0 

2 

1.85 

.02 

1.41 

1.60 

6 

2.19 

.00 

2 

2.13 

.00 

-.06 

1.61 

1.80 

0 

1.81 

2.00 

2 

.00 

.00 

2.01 

2.20 

0 

2.21 

2.40 

0 

2.41 

2.60 

0 

2.61 

2.80 

1 

.00 

2.81 

3.00 

2 

.00 

.00 

**  p<.01 


0 


-41- 


Table  B-14 

Adaptive  and  Conventional  Test  Mean  Information  Values  [!(§}]  a^  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (Q)  for  Subtest  10 


6 Interval 

Adap*"  e Test 

Conventional  Test 

Mean  Difference 

Min 

Max 

N 

lA^) 

a 

S.D. 

N 

I (0) 
c 

S.D.  [I  (6)-I  (0)1  t 

c a 

df 

-3.00 

-2.80 

0 

0 

-2.79 

0 

0 

-2.59 

-2.40 

0 

0 

-2.39 

-2.20 

1 

.00 

0 

-2.19 

-2.00 

0 

0 

-1.99 

-1.80 

11 

.21 

.08 

3 

.30 

.03 

.09 

-1.79 

-1.60 

11 

.51 

.15 

11 

.47 

.09 

-.04 

-.76 

20 

-1.59 

-1.40 

15 

.87 

.28 

8 

1.18 

.17 

.31 

-1.39 

-1.20 

21 

2.07 

.32 

16 

1.89 

.29 

-.18 

-1.76 

35 

-1.19 

-1.00 

15 

3.39 

.54 

19 

3.25 

.51 

-.14 

-.78 

32 

-0.99 

-0.80 

21 

4.84 

1.20 

28 

5.23 

.66 

.39 

1.45 

47 

-0.79 

-0.60 

24 

6.79 

,29 

21 

6.94 

.25 

.15 

1.85 

43 

-0.59 

-0.40 

26 

7.25 

.04 

29 

7.24 

.04 

-.01 

-.93 

53 

-0.39 

-0.20 

30 

7.09 

.04 

32 

7.07 

,04 

-.02 

-1.97 

60 

-0.19 

0.00 

65 

4.02 

3.64 

83 

3.24 

3.64 

-.78 

-1.29 

146 

0.01 

0.20 

22 

8.05 

.27 

28 

7.43 

2.12 

-.62 

-1.36 

48 

0.21 

0.40 

21 

8.56 

.05 

23 

8.59 

.06 

.03 

1.79 

42 

0.41 

0.60 

15 

7.02 

1.98 

14 

7.68 

.44 

.66 

1,22 

27 

0.61 

0.80 

20 

5.61 

.65 

18 

5.66 

.64 

.05 

.24 

36 

0.81 

1.00 

10 

3.89 

.48 

15 

3.88 

.46 

-.01 

-.05 

23 

0.01 

1.20 

11 

2.69 

,20 

11 

2.09 

1.05 

-.60 

-1.86 

20 

1.21 

1.40 

10 

1.94 

.69 

3 

2.14 

.06 

.20 

1.41 

1.60 

2 

2.14 

.11 

3 

2.08 

.01 

-.06 

1.61 

1.80 

1 

.30 

1.81 

2.00 

0 

2.01 

2.20 

0 

2.21 

2.40 

0 

2.41 

2.60 

0 

2.61 

2.80 

0 

2.81 

3.00 

3 

o 

o 

.00 

00 


00 
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Table  B-15 


Adaptive  and  Conventional  Test  Mean  Information  Values  [1(6)1  and  Mean  Difference 
In  Information  and  t Values  at  Estimated  Achievement  Levels  (§)  for  Subtest  11 


6 Interval 

Adaptive  Test 

Conventional  Test 

Mean 

Difference 

Min 

Max 

N 

laCO) 

S.D. 

M 

I (0) 
c 

S.D.  [I  (0)-I 
c a 

(6)1  t 

df 

-3.00 

-2.80 

0 

0 

-2.79 

-2.60 

0 

0 

-2.59 

-2.40 

2 

.04 

.00 

0 

-2.39 

-2.20 

4 

.03 

.03 

0 

-2.19 

-2.00 

16 

.13 

.03 

0 

-1.99 

-1.80 

11 

.25 

.05 

0 

-1.79 

-1.60 

14 

.44 

.15 

0 

-1.59 

-1.40 

21 

.74 

.09 

6 

.76 

.08 

.02 

-1.39 

-1.20 

22 

1.67 

.29 

18 

1.13 

.12 

-.54 

-7.39** 

38 

-1.19 

-1.00 

36 

1.69 

.22 

31 

1.91 

.13 

.22 

4.88** 

65 

-0.99 

-0.80 

22 

2.61 

.28 

34 

2.69 

.30 

.08 

1.00 

54 

-0.79 

-0.60 

25 

3.40 

.74 

52 

3.57 

.27 

.17 

1.47 

75 

-0.59 

-0.40 

33 

4.15 

.12 

32 

4.23 

.09 

.08 

3.03** 

63 

-0.39 

-0.20 

25 

4.27 

.06 

50 

4.28 

.06 

.01 

.68 

73 

-0.19 

0.00 

54 

1.89 

1.95 

63 

1.10 

1.75 

-.79 

-2.31* 

115 

0.01 

0.20 

19 

3.30 

.59 

32 

3.09 

1.02 

-.21 

-.82 

49 

0.21 

0.40 

12 

3.19 

.01 

16 

3.20 

.02 

.01 

1.59 

26 

0.41 

0.60 

11 

3.30 

.05 

11 

3.27 

.05 

-.03 

-1.41 

20 

0.61 

0.80 

8 

2.95 

1.19 

11 

3.39 

.02 

.44 

0.81 

1.00 

9 

3.18 

.09 

1 

3.33 

.15 

1.01 

1.20 

1 

.00 

2 

2.67 

.04 

2.67 

1.21 

1.40 

3 

2.13 

.03 

2 

2.28 

.12 

.15 

1.41 

1.60 

2 

1.98 

.01 

1 

2.00 

.02 

1.61 

1.80 

1 

2.00 

0 

1.81 

2.00 

0 

1 

2.06 

2.01 

2.20 

1 

2.71 

0 

2.21 

2.40 

0 

0 

2.41 

2.60 

0 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

0 

0 

* p<.05 

**  p<.01 
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Tablft  6-16 

Adaptive  and  Conventional  Test  Mean  Infdrmatlon  Values 
In  Inforaatlon  and  t Values  at  Esjfedinated  Achievement 


0 Interval 


Adaptive  Je^t  Conventional  Test 


Mean  Difference 


Min 

Max 

N 

S.D. 

N 

I (6) 
c 

S.D. 

[I  (0)-!  (6)1 
c a 

t 

df 

-3.00 

-2.80 

0 

-2.79 

-l.bO.-' 

0 

0 

-2.59 

-2^K0 

11 

.11 

.32 

0 

-2.39 

20 

7 

.04 

.01 

0 

-2.\y' 

-2.00 

15 

.06 

.03 

0 

-1.80 

13 

.20 

.04 

0 

-1.79 

-1.60 

12 

.41 

.07 

1 

.53 

-1.59 

-1.40 

15 

.88 

.28 

10 

.95 

.IT 

.07 

.71 

23 

-1.39 

-1.20 

23 

1.73 

.24 

21 

J.-81 

.26 

.08 

1.06 

42 

-1.19 

-1.00 

23 

2.63 

.67 

24' 

2.81 

.32 

.18 

1.18 

45 

-0.99 

-0.80 

17 

4.04 

.24 

31 

3.86 

.28 

-.18 

-2.24* 

46 

-0.79 

-0.60 

27 

4.57 

.14 

32 

4.56 

.12 

-.01 

-.30 

57 

-0.59 

-0.40 

33 

4.64 

.83 

44 

4.80 

.01 

.16 

1.28 

75 

-0.39 

-0.20 

23 

4.75 

.02 

35 

4.76 

.02 

.01 

1.86 

56 

-0.19 

0.00 

49 

2.07 

2.37 

80 

2.03 

2.37 

-.04 

-.09 

127 

0.01 

0.20 

19 

4.97 

.09 

24 

4.36 

1.69 

-.61 

-1.57 

41 

0.21 

0.40 

16 

5.23 

.05 

16 

5.23 

.06 

.00 

.00 

30 

0.41 

0.60 

10 

5.25 

.05 

12 

5.27 

.04 

.02 

1.04 

20 

0.61 

0.80 

11 

4.84 

.17 

10 

4.89 

.15 

.05 

.71 

19 

0.81 

1.00 

10 

3.35 

1.77 

9 

4.29 

.16 

.94 

1.01 

1.20 

4 

3.73 

.04 

7 

3.60 

.09 

-.13 

1.21 

1.40 

7 

2.89 

1.28 

5 

3.36 

.05 

.47 

1.41 

1.60 

4 

3.30 

.00 

1 

3.32 

.02 

1.61 

1.80 

1 

3.32 

0 

1.81 

2.00 

1 

3.69 

0 

2.01 

2.20 

0 

0 

2.21 

2.40 

0 

1 

6.66 

2.41 

2.60 

1 

6.67 

0 

2.61 

2.80 

0 

0 

2.81 

3.00 

0 

1 

2.26 

* p<.05 


Estijiated  Information  (1(9)1  Estimated  Information  17(0)) 
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APPENDIX  C 

Supplementary  Figures 


Figure  C-1 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  2 
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Figure  C-2 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  3 
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Figure  C-3 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  4 
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Figure  C-4 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  5 
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Figure  C-5 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  6 
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Figure  C-6 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  7 
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Figure  C-7 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  8 
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Figure  C-8 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  9 
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Figure  C-9 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  10 
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Figure  C-10 

Smoothed  Information  Curves  for 
Adaptive  and  Conventional  Tests  for  Subtest  11 
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Deputy  ADCNO  for  Civilian  Planning 
and  Programing  (Acting) 

Room  2625,  Arlington  Annex 
Washington,  DC  20370 

1 Mr.  George  N.  Graine 
Naval  Sea  Systems  Command 
SEA  047C12 

Washington,  DC  20362 

1 Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 
Attn.  Dr.  Norman  J.  Kerr 

1 Principal  Civilian  Advisor 

for  Education  and  Training 
Naval  Training  Command,  Code  OOA 
Pensacola,  FL  32508 
Attn:  Dr.  William  L.  Maloy 

1 Dr.  Alfred  F.  Siiiode,  Director 

Training  Analysis  4 Evaluation  Group 
Department  of  the  Navy 
Orlando,  FL  32813 

1 Chief  of  Naval  Education  and 
Training  Support  (OlA) 

Pensacola,  FL  32509 

1 Naval  Undersea  Center 
Code  303 

San  Diego,  CA  92132 
Attn;  W.  Gary  Thomson 

1 Navy  Personnel  R40  Center 
Code  01 

San  Diego,  CA  92152 

5 A.  A.  SJoholm,  Head,  Technical  Support 
Navy  Personnel  R4D  Center 
Code  201 

San  Diego,  CA  92152 

2 Navy  Personnel  R4D  Center 
Code  310 

San  Diego,  CA  92152 
Attn:  Dr.  Martin  F.  Wiskoff 


1 Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 
Attn:  Library 


1 Capt.  D.  M.  Gragg,  MC,  USN 

Head,  Section  on  Medical  Education 
Uniformed  Services  Univ.  of 
the  Health  Sciences 
6917  Arlington  Road 
Bethesda,  MO  20014 

1 Dr.  John  Ford 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1 Dr.  Worth  Scanland 

Chief  of  Naval  Education  4 Training 
NAS,  Pensacola,  FL  325C8 

1 Dr.  Richard  A.  Poliak 
Academic  Computing  Center 
U.S.  Naval  Academy 
Annapolis,  HD  21402 

1 N.ivy  Personnel  RSD  Center 
Code  306 

S:''i  Diego,  CA  92152 
AvLn:  Dr.  James  McGrath 

1 Dr.  Leonard  kroeker 

Navy  Personnel  PfO  Center 
San  Oieno.  CA  92152 


1 Technical  Director 

U.S.  Amy  Research  Institute  for  the 
Behavioral  4 Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 Armed  Forces  Staff  College 
Norfolk,  VA  23511 
Attn:  Library 

1 Commandant 

U.  S.  Army  Infantry  School 
Fort  Banning,  GA  31905 
Attn;  ATSH-I-V-IT 

1 Conmandant 

U.  5.  Army  Institute  of  Administration 
Attn;  EA 

Fort  Benjamin  Harrison,  IN  46216 

1 Or.  Beatrice  Farr 

U.S.  Amy  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 Dr.  Frank  J.  Harris 

U.S.  Amv  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 Dr.  Ralph  Ousek 

U.S.  Amy  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 Or.  Leon  Nawrockl 

U.S.  Army  Research  Institute 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 

1 Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Elsenhower  Avenue 
Alexandria,  VA  22333 


Air  Force 


; ,.r.  Ralpn  Canter 

u.S.  Ar.-y  Research  Institute 
SOj'i  Eisennower  Avenue 
A'.exu'.^ria , VA  22333 

1 Or.  James  L.  Raney 

J.S.  Ari;;y  Research  Institute 
bOOl  Eisenhower  Avenue 
Alena nOria,  VA  22333 

1 Or.  Milton  S.  Katz.  Chief 

Individual  Training  & Performance 
Evaluation  Tecnnical  Area 

O. S.  Army  Research  Institute 
bCOl  Eisenhower  Avenue 
Alexandria,  VA  22333 

1 Col.  G.  8.  Howard 
Li . S . Army 

Training  Support  Activity 
Fort  Eustis,  VA  23604 

1 Col.  Frank  Hart,  Director 
Training  Management  Institute 
U.S.  Army,  Bldg.  172s 
Fort  Eustis,  VA  23604 

1 HQ  USARELiE  & 7th  Army 
ODCSOPS 

liSAREOR  Director  of  GED 
APO  hew  York  09403 

1 ARl  Field  Unit  - Leavenworth 

P.  0.  Box  3122 

Ft.  Leavenworth,  KS  66027 

1 DCDR,' USAADMINCEN 
Bldg.  rl , A310 
Attn.  AT21-0ED  Library 
Ft.  Benjamin  Harrison,  IN  46216 

1 Dr.  James  Baker 

U.S.  Army  Research  Institute 
6001  Elsenhower  Avenue 
Alexandria,  VA  22333 

1 Dr.  James  McBride 

Armv  Rpsparrh  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

2 Army  Research  Institute 
Department  of  the  Army 
1300  Wilson  Blvd 
Arlington,  VA  22209 

1 API,  Field  Unit 

ATTN:  Dr.  Donald  Haggard 
Fort  Knox,  KY  40121 

2 Aerospace  Psychology  Dept 
(Code  L5) 

Naval  Aerospace  Med.  Res.  Lab 
Pensacola,  FL  32512 

6 Office  of  Naval  Research 
Code  1021 P (ONRL) 

800  N.  Quincy  Street 
Arlington,  VA  22217 

1 Tng.  Analysis  * Evaluation  Group 
Naval  Mg.  Eguip  Cen-Code  N-OOT 
Orlando,  FL  32813 


1 Research  Branch 
AFMPC/DPMTP 

Randolph  AFB,  TX  78148 

1 AFhRL/AS  (Dr.  G.  A.  Eckstrand) 
wrignt-Patterson  AFB 
Onio  45433 

1 Dr.  Marty  Rockway  (AFHRL/TT) 
Lowry  AFB 
Colorado  80230 

1 Instructional  Technology  Branch 
AFHRL 

Lowry  AFB,  CO  80230 

1 Dr.  Alfred  R.  Fregly 
AFOSR/M,  Sufi  ding  410 
Bolling  AFB,  DC  20332 

1 Dr.  Sylvia  R.  Mayer  (MCIT) 

HQ  Electronic  Systems  Division 
LG  Hanscom  Field 
Bedford,  MA  01730 

1 Air  Force  Human  Resources  Lab 
AFHRL/PED 

Brooks  AFB,  TX  73235 

1 Major  Wayne  S.  Sell  man 
Chief,  Personnel  Testing 
AFMPC/DPMYO 

Randolph  AFB,  TX  78148 


1 Air  University  Library 
AUL/LSE  76-443 
Maxwell  AFB,  Al  36112 


1 Dr.  Donald  E.  Meyer 
U.S. Air  Force 
ATC/XPTD 

Randolph  AFB,  TX  78148 
1 TTOF 

Keesler  AFB,  MS  39534 
1 ATC/TTS 

Randolph  AFB,  TX  78148 

1 HQ  AV/EOCI 

Maxwell  AFR,  AL  36112 

1 SHCS/MSDM  PLATO  IV 
Sheppard  AFB,  TX  76311 

1 ECI/EOXV 

ATTN:  Dr.  Lewiski 
Gunter  AFS,  AL  36118 


Marine  corps 

1 Director,  Office  of  Manpower 
Utilization 

HQ,  Marine  Corps  (Code  MPU) 
BCB,  Building  2009 
Quantico,  VA  22134 

1 Dr.  A.  L.  Slafkosky 
Scientific  Advisor  (Code  RO-1) 
HQ,  U.S.  Marine  Corps 
Washington,  DC  20380 


1 AC/S,  Education  Programs 
Education  Center,  MCDEC 
Quantico,  VA  22134 


Coast  Guard 

1 Mr.  Joseph  J.  Cowan,  Chief 
Psychological  Research  Branch  (G-P-1/62; 
U.S.  Coast  Guard  Headguarters 
Washington,  DC  20590 


Other  DoD 

1 Advanced  Researcn  Projects  Agency 
Administrative  Services 
1400  Wilson  Blvd. 

Arlington,  VA  22209 
Attn:  Ardella  Holloway 

1 Or.  Harold  F.  O'Neil,  Jr. 

Advanced  Research  Projects  Agency 
Cybernetics  Technology,  Room  623 
1400  Wilson  Blvd . 

Arlington,  VA  22209 

1 Dr.  Robert  Young 

Advanced  Research  Projects  Agency 
1400  Wilson  Boulevard 
Arlington,  VA  22209 

1 Mr.  Frederick  W.  Suffa 

Chief,  Recruiting  and  Retention  Evaluation 
Office  of  the  Assistant  Secretary  of 
Defense,  MiRA 
Room  ju970.  Pentagon 
Washington,  DC  20301 

12  Defense  Documentation  Center 
Cameron  Station,  Bldg.  5 
Alexandria,  VA  22314 
Attn:  TC 

1 Military  Assistant  for  Human  Resources 
Office  of  the  Director  of  Defense 
Research  & Engineering 
Room  3D129,  The  Pentagon 
Washington,  DC  20301 

1 Oiri.cor,  Management  Information 
Systems  Office 
OSD,  KARA 

Room  38917,  the  Pentagon 
Washington,  DC  20301 


Other  Govern.nent 

1 Dr.  Lorraine  0.  Eyde 
Personnel  R&D  Center 
U.S.  Civil  Service  Commission 
1900  E Street  NW 
Washington,  DC  20415 

1 Dr.  William  Gorham,  Director 
Personnel  R&D  Center 
U.S.  Civil  Service  Commission 
1900  E Street  NW 
Washington,  DC  2041b 


1 Dr.  Vern  Urry 

Personnel  R&D  Center 
U.S.  Civil  Service  Comnlsslon 
1900  E Street  NW 
Washington,  DC  20415 

1 Dr.  Karshall  S.  Smith 
Associate  Director 
NIE/OPEPA 

National  Institute  of  Education 
Washington,  DC  20208 

1 U.S.  Civil  Service  Commission 
Federal  Office  Building 
Chicago  Regional  Staff  Division 
Regional  Psychologist 
230  S.  Dearborn  Street 
Chicago,  IL  60604 
Attn:  C.  S.  W1n1ew1c2 

1 Dr.  Joseph  L.  Young,  Director 
Memory  & Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 

1 Or.  James  M.  Ferstl 

Employee  Development:  Training 
Tec hnol ogl St 
Bureau  of  Training 
U.S.  Civil  Service  Commission 
Washington,  DC  20415 

1 William  J.  McLaurIn 
Room  301 

Internal  Revenue  Service 
2221  Jefferson  Davis  Hwy. 
Arlington,  VA  22202 


Miscellaneous 

1 Dr.  John  R.  Anderson 
Dept,  of  Psychology 
Yale  University 
Ne«  Haver.,  CT  06520 

1 Dr.  Scarvla  B.  Anderson 
Educational  Testing  Service 
Suite  1040 

3445  Peachtree  Road  NE 
Atlanta,  OA  30326 

1 Professor  Earl  A.  Allulsl 
Code  287 

Dept,  of  Psychology 
Old  Dominion  University 
Norfolk,  VA  23508 

1 Dr.  Daniel  Alpert 

Computer-Based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

1 Ms.  Carole  A.  Bagley 
Applications  Analyst 
Minnesota  Educational 
Computing  Consortluai 
1925  Sather  Ave. 

Lauderdale,  MN  55113 

1 Mr.  Samuel  Ball 

Educational  Testing  Service 
Princeton,  NJ  08540 

1 Or.  Gerald  V.  Barrett 
University  of  Akron 
Dept,  of  Psychology 
Akron,  OH  44325 


1 Dr.  Robert  K.  Branson  , 

lA  Tully  Bldg. 

Florida  State  University 
Tallahassee,  FL  32306 

1 Dr.  John  Seeley  Brown 

Bnit  Beranek  and  Newman,  Inc.  1 

50  Moulton  Street 
Cambridge,  MA  02138 

1 Dr.  Victor  Bunderson 

Institute  for  Computer  Uses  In  Education  ^ 
355  EDLC 

Brigham  Young  University 
Provo,  UT  84601 

1 Or.  Ronald  P.  Carver  ^ 

School  of  Education 
University  of  Missourl-Kansas  City 
5100  Rockhin  Road 
Kansas  City,  MO  64110 

1 Jacklyn  Caselll 

ERIC  Clearinghouse  on  Information 
Resources 

Stanford  University 

School  of  Education  - SCRDT 

Stanford,  CA  94305  1 

1 Century  Research  Corporation 
4113  Lee  Highway 

Arlington,  VA  22207  ^ 

1 Dr.  Kenneth  E.  Clark 

College  of  Arts  & Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

1 Or.  Noman  Cliff 
Dept,  of  Psychology 
University  of  Southern  California 
University  Park 
LOS  Angeles,  CA  90007 

1 Dr.  Allan  M.  Collins 

Bolt  Berunew  ar/l  Netwnan,  Inc. 

50  Moulton  Street 
Cambrioge.  MA  02138 

1 Or.  John  J.  Collins 
Essex  Corporation 
201  N.  Fairfax  St. 

Alexandria,  VA  22314 

1 Or.  Rene  V.  Dawls 
Dept,  of  Psychology 
University  of  Minnesota 
Minneapolis,  MN  55455 

I Dr.  Ruth  S.  Day 

Center  for  Advanced  Study  In 
the  Behavioral  Sciences 
202  Junipero  Serra  Blvd. 

Stanford,  CA  94305 


1 Dr.  John  D.  Carroll 
Psychometric  Lab 
Davie  Hall  01 3A 
University  of  North  Carolina 
Chapel  Hill,  NC  27514 

1 Dr.  Marvin  D.  Dunnette 
Dept,  of  Psychology 
Univerilty  of  Minnesota 
Minneapolis,  MN  55455 

1 ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Bethesda,  MO  20014 


Major  I . N.  Evonic 
Canadian  Forces  Personnel 
Applied  Research  Unit 
1107  Avenue  Road 
Toronto,  Ontario,  CANADA 

Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.  0.  Box  168 

Iowa  City,  lA  52240 

Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organization 

8555  Sixteenth  Stree. 

Silver  Spring,  MD  20910 

Dr.  Larry  Francis 
University  of  Illinois 
Computer-Based  Educational  Research  Lab 
Champaign,  IL  61601 

Dr.  John  R.  Frederlksen 
Bolt  Beranek  A Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

Or.  Vernon  S.  Gerlach 
College  of  Education 
146  Payne  Bldg.  B 
Arizona  State  University 
Tempe,  AZ  85281 

Dr.  Robert  Glaser,  Co-Director 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1 Or.  Richard  S.  Hatch 

Decision  Systems  Assoc.,  Inc. 

5640  Nicholson  Lane 
Rockville,  MD  20852 

1 Dr.  M.  D.  Havron 

Human  Sciences  Research,  Inc. 

7710  Old  Spring  House  Road 
West  Gate  Industrial  Park 
McLean,  VA  22101 

1 Dr.  Duncan  Hansen 
School  of  Education 
Memphis  State  University 
Memphis,  TN  38118 

1 COR  Mercer 

CNET  Liaison  Officer 
AFHRL/Flying  Training  Olv. 

Williams  AFB,  AZ  85224 


1 HumRRO/Western  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 
Attn:  Library 

1 HumRRO/Columbus  Office 

Suite  23,  2601  Cross  Country  Drive 
Columbus,  GA  31906 

1 Or.  Lawrence  B.  Johnson 

Lawrence  Johnson  t Associates,  Inc. 

Suite  502 

2001  S Street  NW 

Washington,  DC  20009 


1 Dr.  Arnold  F.  Kanarlck 
Honeywell.  Inc. 

2600  Ridgeway  Pkwy. 

Minneapolis,  MN  55413 

1 Dr.  Roger  A.  Kaufman 
203  Dodd  Hall 
Florida  State  University 
TaUahasses,  FL  32306 

1 Dr.  Steven  W.  Keele 
Dept,  of  Psychology 
University  of  Oregon 
Eugene,  OR  97403 

1 Dr.  David  Klahr 
Dept,  of  Psychology 
Carnegle-Mellon  University 
Pittsburgh,  PA  15213 

1 Dr.  Ezra  S.  Krendel 
Wharton  School , DH/CC 
Unlv.  of  Pennsylvania 
Philadelphia,  PA  19174 

1 Dr.  Alma  E.  Lantz 
University  of  Denver 
Denver  Research  Institute 
Industrial  Economics  Division 
Denver,  CO  80210 

1 Or.  Frederick  M.  Lord 

Educational  Testing  Service 
Princeton,  NJ  08540 

1 Mr.  Brian  McNally 

Educational  Testing  Service 
Princeton,  NJ  06540 

1 Dr.  Robert  R.  Mackle 

Human  Factors  Research,  Inc. 

6780  Corton  Drive 

Santa  Barbara  Research  Park 

Ooleta,  CA  93017 

I Mr.  Edmond  Marks 

304  Grange  Bldg. 

Pennsylvania  State  University 
University  Park,  PA  16802 

1 Dr.  Leo  Monday 

Houghton  Mifflin  Co. 

, P.  0.  Box  1970 

Iowa  City,  lA  52240 

1 Dr.  Donald  A.  Norman 

! Dept,  of  Psychology  C-009 

University  of  California,  San  Ole 
; La  Jolla,  CA  92093 

I 1 Mr.  Luigi  Petrullo 

; 2431  N.  Edgewood  Street 

! Arlington,  VA  22207 

I 

I ur.  Steven  M.  Pine 

N 660  Elliott  Hall 
University  of  Minnesota 
75  East  River  Road 
Minneapolis,  MN  55455 

1 Or.  Lyman  W.  Porter,  Oean 

Graduate  School  of  Administration 
University  of  California 
Irvine,  CA  92717 


1 Or.  Diane  H.  Ramsey-Klee 
R-K  Research  t System  Design 
3947  Ridgemont  Drive 
Malibu,  CA  96265 


1 R.Oir.  M.  Rauch 
P II  4 

Bur.desmlnisteriuro  der  Verteldigung 
Postfach  161 
53  Bonn  1 , GERMANY 

1 Dr.  Joseph  W.  Rigney 

University  of  So.  California 
Behavioral  Technology  Laboratories 
3717  South  Grand 
Los  Angeles,  CA  90007 

1 Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1 Dr.  Leonard  L.  Rosenbaum,  Chairman 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1 Dr.  Mark  D.  Reckase 

Educational  Psychology  Dept. 
University  of  Missouri -Columbia 
12  Hill  hall 
Columbia.  MO  65201 

1 Dr.  Robert  J.  Seidel 

instructional  Technology  Group, 

Hu.'.lRRO 

300  N.  Washington  St. 

Alexandria,  VA  22314 

1 Dr.  Ricnard  Snow 
Stanford  University 
School  of  Eoucatfon 
Stanford,  CA  94305 

1 Mr.  Dennis  J.  Sullivan 

c/o  Canyon  Research  Group,  Inc. 
32107  Llndero  Canyon  Road 
Westlake  Village,  CA  9136G 

1 Mr.  Walt  W.  Tornow 

Control  Data  Corporation 
Corporate  Personnel  Research 
P.  0.  Box  0 -HON060 
Minneapolis,  MN  55440 

1 Dr.  Benton  J.  Underwood 
Dept,  of  Psychology 
Northwestern  University 
Evanston,  IL  60201 

1 Dr.  Carl  R.  Vest 

Sattelle  Memorial  Institute 
Washington  Operations 
2030  M Street  NW 
Washington,  DC  20036 

1 Dr.  Keith  Wescourt 
Oept.  of  Psychology 
Stanford  University 
Stanford,  CA  94305 

1 Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 


1 Dr.  Thomas  G.  Sticht 

Assoc.  Director,  Basic  Skills 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 


1 Prof.  Fumiko  Samejima 
Dept,  of  Psychology 
Austin  Peay  Hall  304C 
University  of  Tennessee 
Knoxville,  TN  37916 


1 Dr.  Meredith  Crawford 
5605  Mcntgomery  Street 
Cnevy  Cnase,  Mu  20015 

1 Dr.  Nicholas  A.  Bond 
Oept.  of  Psychology 
Sacra.mento  State  College 
6000  Jay  Street 
Sacramento,  CA  95819 

1 Dr.  James  Greeno 
Learning  RiD  Center 
university  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1.  Dr.  Frederick  nayes-Roth 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  CA  90406 

1 Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  1 lA,  Yale  Station 
New  Haven,  CT  06520 

1 Or.  Walter  Schneider 
Oept.  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1 Dr.  Richard  B.  Mi  11  ward 
Dept,  of  Psychology 
Hunter  Lab 
Brown  University 
Providence,  RI  82912 

1 American  Institute  for  Research 
1055  Thomas  Jefferson  St.,  N.W. 
Washington,  D.C.  20007 

1 Applied  Psychological  Services 
Science  Center 
404  East  Lancaster  Ave. 

Wayne,  PA  19087 

1 Bolt,  Beranek,  and  Newman,  Inc. 
50  Moulton  St. 

ATTN:  Library 
Cambridge,  MA  02138 

I Dunlap  Associates,  Inc. 

One  Parkland  Drive 
ATTN:  Library 
Darien,  CT  6820 

1 Educational  TesLlng  Service 
ATTN:  Library 
Rosedale  Road 
Princeton,  NJ  08540 

1 HumRRO/Eastern  Division 
300  North  Washington  St. 

ATTN:  Library 
Alexandria,  VA  22314 


1 Rockwell  International 

Los  Angeles  International  Airport 
ATTN:  8-1  Olv.  TIC  Dept.  299 
Los  Angeles,  CA  90009 


1 System  Development  Corporation 
ATTN.  Technical  Info  Ctr 
Mail  Drop  Al-41 
?500  Colorado  Ave. 

Santa  Monica,  CA  '>(1406 

1 Advanced  Research  Resources  Organization 
8555  Ihth  Street 
Silver  Spring,  MD  20910 

I Colorado  State  University 
Department  of  Psychology 
It.  Collins,  CO  80521 

I University  of  Denver 
Department  of  Psychology 
2030  South  York 
Denver,  CO  80210 

I Florida  State  University 
CAI  Center 
Tully  Building 
Tallahassee,  Ft  32306 

I Dept,  of  Behavioral  Sciences 
University  of  Chicago 
5848  S.  University  Ave. 

Chicago,  IL  60601 


1 Illinois  State  University 
ATT’.:  Document  Librarian 
lJci~"al , IL  61761 

I Oocments  Division 
University  of  Illinois 
ATTN:  Library 
Urbana,  IL  47907 

I Department  of  Psychology 
Purdue  University 
Lafayette,  IN  47907 

I Department  of  Psychology 
University  of  Houston 
Houston,  TX  77004 

1 Dr.  Danile  Alpert 
Computer-Based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61820 

2 Or.  Ernest  J.  Anastasio 
Associate  Director 

Data  Analysis  Research  Division 
Educational  Testing  Service 
Princeton,  NJ  08540 

I Mr.  Ayron  B.  Barr 
19  Ventura  Hall 
I.M.S.S.S. 

Stanford  University 
Stanford,  CA  94305 

1 Or.  Arthur  S.  Blaiwes 

Naval  Training  Equipment  Center 
Cciue  N 215 
Orlando,  FL  32813 

1 Dr.  Victor  Bunderson 
Brigham  Young  University 
Institute  for  Computer  Uses  in  Ed. 
W 164  STAD 
Provo,  UT  84601 

1 Or.  Polly  Carpenter-Huffman 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  CA  90401 


1 Or.  Jeanne  S.  Chall 

Graduate  School  of  Education 
Harvard  University 
Roy  E . Larsen  Hal  1 
Appian  Hay 

Cambridge,  MA  02138 

1 Or.  Irene  Clements 
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